Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning

We thank reviewer 9Cf7 for the thoughtful and detailed feedback. We appreciate your recognition of our theoretical framework, the clarity of Theorem 6, and the relevance of DAgger-style analysis to modern robotics. Below, we address your specific concerns.

For W1 and Q1: Applicability of Theoretical Results to Continuous Control

We thank the reviewer for this insightful observation. We agree that our theoretical analysis is specific to discrete-action MDPs and does not directly extend to continuous control settings such as MuJoCo. Our intention was to use MuJoCo primarily to illustrate the practical viability of our state-wise annotation model and algorithms, rather than to validate our theoretical bounds. We agree that continuous imitation learning presents unique challenges such as exponential-in-horizon compounding errors, and that DAgger-style theoretical analysis in this setting remains largely open.

We will revise relevant statements to clarify this distinction:

"Empirical results on the synthetic MDP support our theoretical findings, while MuJoCo experiments demonstrate the practical viability and competitive performance of our state-wise annotation model and algorithms on continuous control tasks."

That said, we are currently conducting additional experiments on discrete-action environments (e.g., Atari) and language model distillation tasks, which more directly align with our theoretical framework. Results from these ongoing efforts will be included in the final or extended version of the paper.

For W2: Probabilistic Switching vs. Warm-Start

For clarity of exposition, we analyze DAgger without stochastic mixing with the expert (as also done in [1]), since setting $\beta = 0$ yields the best policy suboptimality guarantee in the original DAgger paper. We agree that stochastic mixing with the expert is a useful practical heuristic. Meanwhile, Warm-Stagger can also be viewed as a form of stochastic mixing with $\beta = 1$ for the $N_{\text{off}}$ offline trajectories and $\beta = 0$ for interactive rounds. While properly tuned probabilistic switching may also mitigate warm-start issues in our MDP example, we adopt this hybrid IL formulation to enable a clean analysis. Extending our framework to incorporate stochastic mixing is an interesting direction for future work.

For Q2: Clarification of Terminology – "Offline", "Interactive" , "Hybrid"

We appreciate the reviewer’s suggestion. In this paper, we deliberately use the terms "offline", "interactive", and "hybrid", to distinguish between three supervision regimes. The term online only appears in the context of online learning as induced by our theoretical reduction, and we deliberately avoid using online imitation learning to prevent confusion.

Thank You!

We hope the above clarifications address your concerns, and we sincerely thank you for your thoughtful and supportive feedback.

[1] D. J. Foster, A. Block, and D. Misra. “Is behavior cloning all you need? Understanding horizon in imitation learning.” NeurIPS 2024.

We are glad that reviewer yovM found our work “theoretically sound” and appreciated that the “experiments also show good results.” Below we address specific concerns:

For W: Practical Significance and Baseline Design

We appreciate the thoughtful question of the reviewer. Our work is motivated by the need to understand and formally analyze warm-start and non–full-trajectory annotation strategies in interactive imitation learning, despite their widespread use in practice (e.g., [3,4]). As noted by Reviewer 9Cf7, DAgger-style algorithms are fundamental to modern data pipelines for robotics (e.g., [6,7]).

As noted in Appendix G, we train a single parametric MLP using log-loss with empirical risk minimization (ERM), retraining the policy after each newly annotated state. Thus, in our experiments our method does not initialize a small set of predefined policies. We apologize for the confusion, and will clarify this more explicitly in the main experimental section.

We interpret the reviewer's suggestion of "test set logic" baseline as behavior cloning (BC), which we already benchmark. Please let us know if we misunderstood the suggestion.

For Q1 and Q2: Scalability of Policy Class and Experimental Setup

We use a single parametric MLP policy in MuJoCo experiments, trained via log-loss with ERM. There is no predefined finite policy class or mixture of policies. In our synthetic MDP experiment, where the policy class scales combinatorially (e.g., $B = 100^{1200}$ in Figure 2), we simulate the effect of a randomized mixture by assigning random actions to previously unseen states.

For Limitation: Realizability Assumption, Environment Cost Modeling, and Expert Design

We agree that we should be aware of the suboptimality of the expert in practical deployment of imitation learning; in some applications it may be preferable to combine imitation and reinforcement learning signals (e.g., [8]). We will add this discussion in a "limitation" section the final version. Modeling environment query cost is an interesting direction, which we intentionally omitted for clarity of presentation (see line 301), and aim to explore in future work.

Thank You!

If the above clarifications address your concerns, we would greatly appreciate a reconsideration of the score.

[3] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. “HG-DAgger: Interactive imitation learning with human experts.” ICRA 2019.

[4] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. “ThriftyDagger: Budget-aware novelty and risk gating for interactive imitation learning.” CoRL, 2022.

[6] L. X. Shi et al. “Yell at your robot: Improving on-the-fly from language corrections.” RSS, 2024.

[7] Y. Jiang, C. Wang, R. Zhang, J. Wu, and L. Fei-Fei. “TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction.” CoRL, 2025.

[8] S. Ross and J. A. Bagnell. “Reinforcement and imitation learning via interactive no-regret learning.” arXiv, 2014.

We thank reviewer rsrZ for the thoughtful and constructive review. We appreciate your recognition of our theoretical contributions and your positive evaluation. Below we address specific concerns:

For W1: Worst-Case Bounds and Instance-Wise Comparisons

We acknowledge the reviewer’s valid observation that Theorems 2–4 provide worst-case upper bounds, which do not imply generic instance-wise superiority. We are glad that Theorem 6 was seen as a meaningful step toward closing this gap, and we appreciate the reviewer’s recognition of our efforts to go beyond worst-case analysis.

For W2: Applicability of Continuous Control Experiments to Support Theoretical Bounds

Indeed, our theoretical guarantees apply only to discrete-action MDPs. The MuJoCo experiments are intended to empirically evaluate the practical effectiveness of the proposed algorithms, not to validate the theory. We will revise related statements to: "Empirical results on the synthetic MDP support our theoretical findings, while MuJoCo experiments demonstrate the practical viability and competitive performance of our methods on continuous control tasks." to accurately reflect this distinction.

We agree that continuous imitation learning presents unique challenges such as exponential-in-horizon compounding errors, and that DAgger-style theoretical analysis in this setting remains largely open, which is an interesting direction. We will explicitly acknowledge this in the experiment section.

Our analysis is based on log-loss, which reduces to the indicator loss under deterministic experts in discrete-action settings. While indicator losses are indeed incompatible with continuous action spaces, log-loss remains applicable. Our implementation employs log-loss for continuous control tasks (see Appendix G, Line 1454), consistent with the Walker2d-v4 experiment setting with logarithmic loss in Foster et al. (2024) [1]

For Minor Comments

1. Inconsistent use of ln vs. log. We will use log consistently in the final version.

2. Line 210: equality vs. inequality. Though the equality is technically correct under $\mathcal{O}(\cdot)$, we will revise it to $\leq$ for consistency in presentation.

3. Loss used in Theorem 2. We will clarify that Theorem 2 establishes a new horizon-independent generalization bound under log-loss, distinct from standard 0-1 loss analyses in BC.

For Q: When Does Warm-Stagger Provably Outperform BC and Warm-DAgger?

We appreciate the reviewer’s thoughtful question. To clarify, our work is motivated by the need to understand and formally analyze warm-start and non–full-trajectory annotation strategies in interactive imitation learning, despite their widespread use in practice (e.g., [3,4]).

To the best of our knowledge, we are the first to formally analyze hybrid imitation learning that combines warm-start with interactive annotation. Our analysis framework is also transferable and can be extended to analyze other DAgger-style algorithms initialized from offline data.

We are not exactly sure which version of the Warm-DAgger algorithm the reviewer is referring to; thus we present and discuss three possible interpretations below:

(1) Naively changing Warm-Stagger (Algorithm 2) to query full trajectory-wise annotation at step 5.

This variant warm-starts with offline data and, in each interaction round, executes an on-policy rollout and retroactively labels the entire trajectory. Let $M_{\text{on}}$ be the number of such trajectories.

Following similar analysis, the suboptimality is bounded by $J(\pi_E) - J(\hat{\pi}) \leq \mathcal{O}( \min ( \frac{R \log B}{N_{\text{off}}}, \frac{\mu H \log B}{M_{\text{on}}})),$

which matches the Warm-Stagger bound but replaces $N_{\text{on}}$ with $M_{\text{on}}$, incurring an extra factor of $H$ when measuring the cost using the number of state-wise annotations.

(2) Adding a warm-start procedure to the LogLossDAgger in [1] (see their Proposition 2.2 and Appendix C.2).

Their guarantee under a deterministic expert is:

$J(\pi^*) - J(\hat{\pi}) \leq \mu \cdot \sum_{h=1}^H \frac{\log(|\Pi_h| H \delta^{-1})}{n}.$

Translating into our notation, with a fixed policy class of size $B$ and $M_{\text{on}}$ on-policy full-trajectory queries, this becomes:

$J(\pi_E) - J(\hat{\pi}) \leq \mathcal{O}(\frac{\mu H \log B}{M_{\text{on}}} ).$

Following similar analysis with a warm-start initialization, the suboptimality of warm-start LogLossDAgger is:

$J(\pi_E) - J(\hat{\pi}) \leq \mathcal{O} ( \min ( \frac{R \log B}{N_{\text{off}}}, \frac{\mu H \log B}{M_{\text{on}}} )),$

which also incurs an extra factor of $H$ in state-wise annotation cost.

(3) Adding a warm-start procedure to our Algorithm 3, Tragger.

By our Theorem 26 (which achieves a better guarantee than [1] and does not require recoverability), under a deterministic expert, Tragger satisfies:

$J(\pi_E) - J(\hat{\pi}) \leq \mathcal{O} \left( \frac{R \log B}{M_{\text{on}}} \right).$

With a warm-start initialization using $N_{\text{off}}$ offline trajectories, the suboptimality becomes:

$J(\pi_E) - J(\hat{\pi}) \leq \mathcal{O} (\min (\frac{R \log B}{N_{\text{off}}}, \frac{R \log B}{M_{\text{on}}} )).$

This is, to our knowledge, the best known guarantee for warm-start DAgger under log-loss with offline initialization and trajectory-wise annotations. However, since $\mu \leq R$, this implies $\mu H \leq R H$, and therefore Warm-Stagger remains more sample-efficient under state-wise annotation.

In all, under our analysis, given a fixed target policy suboptimality Warm-Stagger achieves better state-wise annotation cost than all three warm-start DAgger variants we consider. Therefore, if the considered variants align with the reviewer’s intended formulation, our analysis shows that sparse corrections are provably more efficient.

A final remark to intuitively understand when full-trajectory expert annotation becomes wasteful compared to sparse corrections: consider MDPs with self-absorbing states (as in the lower bounds of [1,5]). In such settings, annotating all $H$ identical states in a trajectory provides no more information than annotating a single state.

Thank You!

We would be happy to include empirical comparisons of Warm-start DAgger variants (specifically, variants (1) and (3)) on MuJoCo benchmarks in the appendix, to further support our theoretical conclusions. If the above response addresses your concerns, we would sincerely appreciate a reconsideration of the score.

[1] D. J. Foster, A. Block, and D. Misra. “Is behavior cloning all you need? Understanding horizon in imitation learning.” NeurIPS 2024.

[3] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. “HG-DAgger: Interactive imitation learning with human experts.” ICRA 2019.

[4] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. “ThriftyDagger: Budget-aware novelty and risk gating for interactive imitation learning.” CoRL, 2022.

[5] N. Rajaraman, Y. Han, L. Yang, J. Liu, J. Jiao, and K. Ramchandran. “On the value of interaction and function approximation in imitation learning.” NeurIPS 2021.

We thank reviewer p7ue for the kind and supportive review. We are glad you found the work a solid contribution both theoretically and experimentally.

For Q1: On the Applicability of Theoretical Bounds to MuJoCo

We agree that our theoretical analysis is limited to discrete MDPs and does not directly extend to continuous control. Our intention was to use MuJoCo experiments to illustrate practical viability of our state-wise annotation model and algorithms rather than to validate theoretical bounds.

We will revise related statements to: "Empirical results on the synthetic MDP support our theoretical findings, while MuJoCo experiments demonstrate the practical viability and competitive performance of our state-wise annotation model and algorithms on continuous control tasks." to accurately reflect this distinction.

That said, we are currently conducting additional experiments on discrete-action environments (e.g., Atari) and language model distillation tasks, which more directly align with our theoretical framework. Results from these ongoing efforts will be included in the final or extended version of the paper.

In MuJoCo, the exact values of $R$ and $\mu$ may be hard to estimate due to the infinite size of the action space. The value of $R$ can be estimated by the cumulative reward of the expert policy over multiple episodes, while $\mu$ can be estimated by replacing a single expert action during a rollout and measuring the maximum performance degradation compared to the original expert. We will include these empirical estimates in the final version under the experimental details section.

For Q2: On Assumptions of Expert Being Realizable and Deterministic

Theoretically, realizable and deterministic expert assumptions enable our favorable $\mathcal{O}(1/N)$-style policy suboptimality guarantees.

Under stochastic experts, we haven't worked out the guarantees of interactive imitation learning yet; we expect that the suboptimality guarantee includes a variance term that vanishes at a rate of $\mathcal{O}(1/\sqrt{N})$ due to the expert’s randomness (analogous to [1]).

Under nonrealizable experts, we expect the suboptimality guarantee of output policy to have two parts:

(1) approximation error measuring how well the policy class approximates the expert;

(2) an optimization error term that typically degrades to $\mathcal{O}(1/\sqrt{N})$ in the agnostic setting (see [2]). Extending the theory to the nonrealizable and stochastic expert settings is an interesting direction for future work.

Empirically, we evaluate our algorithm with realizable stochastic expert. Though the nonrealizable setting is beyond the scope of this work, we expect that some variant of our algorithm can still give reasonable performance, provided the policy class is expressive enough (so that the problem is not exactly realizable but still meaningful). For example, [2] observed that with nonrealizable stochastic experts, DAgger variants outperform BC, and exhibit learning curves similar to our Figure 1.

Thank You!

We appreciate the thoughtful questions and hope our clarifications address the concerns raised.

[1] D. J. Foster, A. Block, and D. Misra. “Is behavior cloning all you need? Understanding horizon in imitation learning.” NeurIPS 2024.

[2] Y. Li and C. Zhang. “Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms.” ICML 2024.