Faithful Dynamic Imitation Learning from Human Intervention with Dynamic Regret Minimization

W1 & Q1: Novelty Clarification

We clarify our key contribution to the community of human-in-the-loop imitation learning (HIL-IL) are:

This is the first work to formulate HIL-IL as an online non-convex learning problem with dynamic regret as the optimization objective. Existing HIL-IL methods either use static regret or treat all imitation data indiscriminately. We are the first to identify the problem of shifting data distributions from evolving novice policy, which has been totally overlooked in HIL-IL.

We are also the first to introduce the faithful imitation objective for leveraging hybrid data from both experts and the learning policy. Existing methods (e.g., HACO, IWR, PVP) optimize on mixed novice-expert data without distinction, hindering performance by causing the agent to imitate suboptimal novice actions.

We solve the dynamic regret problem using FTPL-D+ to adapt the dynamic non-convex loss and encode the faithful imitation objective into an unbiased proxy reward using DICE. These two techniques can be replaced with other methods (if any) as long as they can slove dynamic regret problem with non-convex loss and can optimize the unbiased imitation objective.

In addition, it is true that using a discriminator to estimate a density ratio and using weighted behavior cloning to extract policy is a common technique in IL. However, the key difference is how the weights are computed. ILID [1] (mentioned by the reviewer) trains a discriminator to identify expert states and selects preceding actions based on trajectory backtracking. It requires access to contiguous trajectories, while in our human intervention data consists of fragmented trajectories from experts.

W2 & Q2: Rationale of using FTPL-D+ & Theoretical Analysis

The FTPL-D+ algorithm is an extension of the FTPL method. While FTPL is effective for optimizing static regret, it is not suited for dynamic distributions where environmental variation is unknown, a typical scenario in online imitation learning problems.

FTPL-D+ addresses this challenge by using an ensemble learning framework. It maintains a collection of learners, each associated with a different interval parameter, and partitions the time horizon to restart FTPL periodically. A meta-algorithm, Hedge, is then used to adaptively select among these learners. This enables the method to adapt to environmental changes and achieve a strong regret bound $\mathcal{O}\left(T^{2/3}(V_T + 1)^{1/3}\right)$, thereby being able to solve the online IL problems.

We will provide the following theoretical analysis on regret guarantees. Our theoretical analysis shows that our algorithm is no-regret and can eventually converge to the optimal policy：

(Rendering issues with long equations led us to split them and add new notations (e.g., $k = i-1$), slightly reducing readability.)

We aim to prove that our FaithDaIL is no-regret, i.e.,

$\frac{1}{M} \mathbb{E}[\mathcal{R}_D] \to 0$

We first give some assumption:

• Bounded Parameter Space: The policy parameter space is effectively bounded due to regularization and optimizer constraints in practice.
• Lipschitz Loss: The imitation loss $\ell_i(\pi)$ is Lipschitz in $\theta_\pi$, assuming (1) $\pi(a|s)$ is Lipschitz, (2) importance weights $\omega_i^\star(s,a)$ are bounded, and (3) $\pi(a|s)$ is bounded away from zero.

Following the analysis from FTPL-D+, the expected dynamic regret of FaithDaIL satisfies:

$\mathbb{E}[\mathcal{R}_D] \le \mathcal{O}\left( M^{\frac{2}{3}} (V_M + 1)^{\frac{1}{3}} \right),$

where:

• $M$ is the number of rounds,
• $V_M = \sum_{i=2}^{M} \max_{\pi \in \mathcal{K}} |l(\pi, \mathcal{D}_i^B, \mathcal{D}_i^H) - l(\pi,\mathcal{D}_k^B, \mathcal{D}_k^H)|$, $k=i-1$ is the total variation of the loss sequence.

To establish that the dynamic regret is sublinear, it suffices to show that $V_M$ grows sublinearly with $M$. We formalize the implicit intervention rate $\beta_i$ and state the core assumption underlying its use:

• Definition (Implicit Intervention Rate $\beta_i$) :

$\beta_i = \mathbb{E}_{\pi^N_i } \left[ \frac{|\mathcal{D}_i^{H,\text{new}}|}{|\mathcal{D}_i^{H,\text{new}}| + |\mathcal{D}_i^N|} \right]$

• Assumption (Intervention Rate Decay) : This reflects the intuition that as the agent improves, fewer human interventions are needed over time.

$\lim_{i \to \infty} \beta_i = 0$

To relate $\beta_i$ to the trajectory distribution shift, we invoke the following result (adapted from DAgger, Lemma 4.1):

• Lemma (Distribution Divergence Bound): With $T$ as the MDP horizon,

$\|d_i^B - d_i^N\|_1 \le 2T\beta_i$

We now bound the single-step loss variation $\Delta_i = \max_{\pi} | \ell_i(\pi) - \ell_{i-1}(\pi) |$.

$|\ell_i(\pi) - \ell_{i-1}(\pi)| \leq |A - B| + |B - C|$

where $k=i-1$ :

$A = \mathbb{E}_{(s,a) \sim d_i^B}[w_i^\star(s,a) \log \pi(a|s)]$

$B = \mathbb{E}_{(s,a) \sim d_k^B}[w_i^\star(s,a) \log \pi(a|s)]$

$C = \mathbb{E}_{(s,a) \sim d_k^B}[w_k^\star(s,a) \log \pi(a|s)]$

1. Term 1 (due to distribution shift):：

$\left| A-B \right| = \left| \int_{(s,a)\in\mathcal{S}\times\mathcal{A}}[d_{i}^B(s,a) - d_{k}^B(s,a)] w_i^\star(s,a) \log \pi(a|s) ds da \right|$

$\leq N_1 \cdot N_2 \cdot \|d_i^B - d_k^B\|_1$

where:

$N1 = \|w_i^\star \|_{\infty}$

$N2 = \|\log \pi\|_{\infty}$

2. Term 2 (due to weight shift):：

$\left| B-C \right| = \left| \int_{(s,a)\in\mathcal{S}\times\mathcal{A}}[w_{i}^\star(s,a) - w_{k}^\star(s,a)] d_k^B(s,a) \log \pi(a|s) ds da \right|$

$\leq N_2 \cdot N_3 \cdot \|w_{i}^\star - w_{k}^\star\|_1$

where:

$N2 = \|\log \pi\|_{\infty}$

$N3 = \|d_k^B \|_{\infty}$

Step 1: Bounding distribution shift: Using triangle inequality and Lemma (Distribution Divergence Bound):

$\|d_i^B - d_k^B\|_1 \le 2T(\beta_i + \beta_k) + \|d_i^N - d_k^N\|_1$

Assuming the novice policy changes smoothly:

$\sum_{i=2}^M \|d_i^N - d_{i-1}^N\|_1 = o(M)$

and $\beta_i \to 0$, we have:

$\sum_{i=2}^M \|d_i^B - d_{i-1}^B\|_1 = o(M)$

Step 2: Bounding weight shift: Assuming that the variation in importance weights is primarily induced by the shift in behavior distributions, and that $V_i^\star$ evolves smoothly under $d_i^B$, we have:

$\|w_i^\star - w_{i-1}^\star\|_1$

$\le L_w \cdot \|d_i^B - d_{i-1}^B\|_1 + o(1)$

Thus:

$\sum_{i=2}^M \|w_i^\star - w_{i-1}^\star\|_1 = o(M)$

Then under the earlier Lipschitz assumptions, we assume relevant terms of N1, N2, N3 are uniformly bounded.

Conclusion: Sublinear functional variation.

$V_M = \sum_{i=2}^M | \ell_i(\pi) - \ell_{i-1}(\pi) | = o(M)$

Final Result: No-regret Guarantee

From FTPL-D+ analysis:

$\mathbb{E}[\mathcal{R}_D] = \mathcal{O}(M^{2/3}(V_M + 1)^{1/3}) = o(M), \quad \frac{1}{M} \mathbb{E}[\mathcal{R}_D] \to 0$

Hence, FaithDaIL is a no-regret algorithm in non-convex dynamic environments, with the learned policy converging to the round-wise optimal.

W3 & Q3: Applicability to Other HIL Settings

We will add a section of discussion about applicability to other HIL settings as follows.

The core challenges FaithDaIL addresses—data efficiency and the impact of suboptimal novice data—are prevalent in many safety-critical domains beyond autonomous driving.

Robotic Manipulation: In tasks like robotic arm manipulation, a human teleoperator may intervene to prevent errors. These interventions, coupled with the robot's evolving policy and potential reaction delays, create the exact type of dynamic, mixed-quality dataset that FaithDaIL is designed to handle. We plan to extend our work to this area using simulators to tackle tasks such as pick-and-place and object manipulation.

Dialogue Systems: High-stakes dialogue systems for customer service or medical consultation also rely on human oversight. An operator may correct or override the system's responses, leading to a mix of optimal and suboptimal conversational paths. As the dialogue agent learns and its response distribution shifts, FaithDaIL's dynamic regret framework can be applied to improve its performance and reduce the need for intervention over time.

Q4: Time Window Sensitivity

FTPL-D+ employs an ensemble of base learners with different time windows ($\tau_k = 2^{k-1}$ where $k$ denotes the $k$-th learner) and leverages a Hedge-style meta-algorithm to adaptively adjust their weights based on performance. This design inherently mitigates the sensitivity problem to different window settings. In practice, we observe that during training, the meta-algorithm tends to assign more weight to policies with shorter time windows in later rounds, suggesting that the method self-adjusts to favor recent, high-quality behavior distributions.

Q5: Faithful Imitation Definition

"faithful imitation" refers to explicitly imitating the human expert's policy, even when using a mixed dataset of expert and novice trajectories for efficiency. This is accomplished through our imitation objective (Equations 2-3), which optimizes the KL divergence towards the human policy distribution. We will formally define this concept in the next version of the paper.

Q6: Discussion on Limitation

We will discuss limitations as follows:

• Evaluation Scope: Validate the framework on other safety-critical platforms, such as robotic arms, by collecting and evaluating data from human teleoperator interventions.
• Expert Variability: Improve the model's robustness to diverse and potentially suboptimal interventions from different human experts. We can learn weight for expert data based on individual performance.
• Intervention Modality: Generalize the framework to support multi-modal feedback (e.g., verbal commands, gestures) instead of only direct control. This would involve encoding varied inputs into a consistent format for the learning loop.

W1: Computational Overhead

We agree that an ensemble of K policies increases computational overhead. However, based on empirical observations from our implementation, we propose two practical strategies to manage this cost.

First, in practise, we observe that the agent's policy distribution progressively converges toward the expert policy distribution, and the best-performing policies within the ensemble are usually those from recent rounds with shorter time intervals. This suggests that the full ensemble, which is required by the theoretical formulation, may not be necessary in practice. The computational cost can be significantly reduced by maintaining only a smaller subset of learners focused on more recent data, without a major impact on performance.

Second, to address scalability as the training horizon (M) grows, the process can be partitioned into sequential stages in practice. Each stage runs for a shorter horizon and "warm-starts" using the best policy from the previous one. This approach prevents the ensemble size from growing indefinitely across a very long training process. We plan to evaluate this staging strategy in future work.

W2: Adversarial Training Instability

We thank the reviewer for this important question. We agree that training stability with a discriminator is a key concern, and we designed FaithDaIL's framework to be fundamentally different from adversarial approaches like GAIL or GANs.

Our method avoids a direct min-max game by decoupling the optimization of the policy and the discriminator. At each round, the discriminator is first trained to help compute a proxy reward and importance weights. The policy is then updated via weighted behavior cloning to imitate behaviors with higher weights, rather than trying to fool the discriminator. This sequential and non-adversarial training paradigm, inherited from the DICE family of algorithms, is inherently more stable [1]. Furthermore, the FTPL-D+ ensemble provides an additional layer of robustness by adaptively managing multiple learners, which naturally mitigates the impact of any single, potentially volatile, discriminator estimation. We will revise Section 4.1 to better clarify this decoupled update process.

[1] DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations (2022 ICLR)

W3: Explanation of FTPL-D+

We thank the reviewer for the helpful suggestion. We will add the following explanation of the FTPL-D+ algorithm in Section 4.2.

FTPL-D+ is an ensemble variant of Follow-the-Perturbed-Leader (FTPL), designed to handle non-stationary characteristics where the loss landscape changes over time. It maintains a set of $K$ FTPL learners, each associated with a different rolling window size (e.g., covering the past $h$ rounds), to capture different temporal dynamics. This mechanism implicitly smooths out the non-stationarity of the data distribution and allows the overall strategy to adapt to varying timescales of change. Such a multi-interval ensemble is particularly well-suited for minimizing dynamic regret, as it balances short-term reactivity (via short-horizon learners) and long-term stability (via long-horizon learners). Theoretical results of FTPL-D+ [2] show that this structure achieves near-optimal dynamic regret bounds in general non-convex settings.

[2] Online non-convex learning in dynamic environments (2024 NIPS)

Q1：Generalization to RL

We thank the reviewer for this insightful question. We think that the core ideas of our FaithDaIL -- optimize dynamic regret to address non-stationarity distribution --- could benefit in a range of non-stationary RL environments, including：

• Non-stationary dynamics, due to environmental changes, multi-agent interactions, or sim-to-real domain shifts: One possible approach is to handle non-stationary environmental changes by formulating an objective to evaluate the accuracy of the dynamic transition model within the dynamic regret framework, which helps the learner adapt to shifts in the environment's underlying dynamics.
• Reward function drift, as seen in preference-based or feedback-driven learning: we can solve the non-stationary reward function shift by defining the expected cumulated reward as the objective of dynamic regret and solving it using a method like FTPL-D+
• Evolving tasks in continual reinforcement learning: we can define a sequence of task-specific loss functions, where each loss may reflect metrics like task completion performance, and incorporate them into the dynamic regret framework accordingly.

Q2: Rationale of using FTPL-D+

We clarify the motivation of using FTPL-D+ as follows. In the domain of online non-convex learning, previous approaches to minimizing dynamic regret often rely on prior knowledge about the dynamics of the loss landscape or require computationally intractable sampling oracles. Therefore, they cannot be used in our HIL-IL problem due to the unknown dynamics.

In contrast, FTPL-D+ does not depend on such prior knowledge, and its underlying offline optimization oracle is much more practical—stochastic gradient descent can serve as an effective approximation in practice and often leads to near-global optima. Given these advantages, we believe FTPL-D+ is currently the most suitable and only choice for addressing our problem setting.

We clarified the computational overhead may not be a issue in practice (see Respone to W1).

W1: Figure Caption

Thanks for the comments. We will add the following descripion in the caption for interpretability.

We can see that both HG-DAgger and IWR fail in senarios with both static and dynamic obstacles (deviates off-road in (b) and (e), collides in (a) and (f)), and have erratic trajectories in curving roads (c) and (d). HACO performs even worse, shows unstable behavior like wide swings in (c), (d), and (e) and lane departures in (a) and (b). PVP showed risky maneuvers to avoid obstacles, resulting in collides in (f) and deviates in (a), (b), and (e). In contrast, FaithDaIL produces the smoothest and safest trajectories, closely aligning with expert driving across all scenarios.

W2: Details on HIL Experiments

We provided the HIL experiment details in Appendix C. To improve clarity, we reorganize and revise the experiment details here. The described Human-in-the-Loop (HIL) procedure was used to collect data for HG-DAgger and all baseline methods in the same manner.

Human Subject Recuitment: Three college students (aged 20–25), each holding a valid driver’s license and with at least two years of driving experience, voluntarily participated in the human-in-the-loop experiments. Participants were recruited through campus-wide advertisements and selected based on their availability, driving experience, and willingness to engage in interactive training tasks. We ensured full transparency by clearly explaining the purpose of the study, the nature of their involvement, and how the collected data would be used. Each participant signed a written informed consent form, indicating their full understanding and voluntary agreement. The study protocol was reviewed and approved by the university Institutional Review Board (IRB), ensuring compliance with ethical standards for research involving human subjects.

HIL Experiments Procedure: Before commencing the experiments, all participants received detailed instructions and underwent a brief training session to familiarize themselves with the simulator interfaces and intervention mechanisms. They practiced for approximately 30 minutes until achieving proficiency, which was defined as attaining $\ge 95\%$ task success over 50 consecutive episodes. In the main human-in-the-loop (HIL) sessions, each participant supervised a learning agent, intervening via a keyboard in MetaDrive or a Logitech G923 racing wheel in CARLA. The objective is to navigate the vehicle safely to the destination. Participants are encouraged to intervene whenever they sense potential danger or behavior misaligned with human norms. To mitigate potential order bias, the presentation sequence of algorithms (for experiments in MetaDrive), simulators, and task scenarios was randomized for each participant. Each human-in-the-loop (HIL) data collection session last approximately one hour (70 minutes for tasks in MetaDrive and 60 minutes for tasks in CARLA). Consequently, each participant dedicated a total of 5 hours to the MetaDrive experiments (which involved interacting with 5 different algorithms). For the CARLA simulator, each participant engaged with our FaithDaIL method for one 60-minute session; performance data for other baseline methods on CARLA were adopted from the findings reported in PVP.

W1: Justification for Dynamic Regret

We clarify that our problem is not "imitation learning with online expert annotation". We study human-in-the-loop imitation learning with active human intervention, in which, human experts supervise the agent's execution, and intervene to make corrections once errors occur. The key difference of our human intervention from "online expert annotation" is that we learn from a hybrid of expert trajectories as well as the changing agent-generated trajectories (which we call novice data), while "online expert annotation" assumes stationary sample demonstrations. This changing distribution of imitation data, is the key reason why we adopt dynamic regret. In contrast, DAgger uses static regret because its data is consistently sampled from the fixed human expert policy distribution.

In many classical dynamic‑regret formulations, the data distribution and loss function or environment dynamics (summarized as loss landscape) change over time. As a result, the optimal policy that minimizes instantaneous loss often varies from one round to the next. However, we argue dynamic regret is designed for problems with non-stationary loss landscape. Whether the optimal policy changes over time is not the determining factor for using dynamic regret. [1] studied dynamic regret when optimal policy remains unchanged but loss or data distribution changes. In this setting, the changing loss landscape introduces complexity beyond what is captured by static regret.

Therefore, although our faithful imitation objective (Section 4.1) assumes that the theoretical optimal policy is fixed, we can still formulate our problem as a dynamic regret problem because of the complexity induced by the changing distribution of imitation data.

[1] On Online Optimization: Dynamic Regret Analysis of Strongly Convex and Smooth Problems (AAAI 2021)

W2: Link to expert reaction delay

We thank the reviewer for this comment. We will clarify the connection in the revision.

Expert reaction delay creates suboptimal "deviation trajectories" where the agent acts incorrectly just before a human intervenes. Learning directly from this data leads to unfaithful imitation. To address this, our faithful imitation objective optimizes the KL divergence between the agent's policy and the expert's policy. Our method assigns lower proxy rewards to suboptimal trajectories, which in turn gives them lower weights during the weighted behavior cloning update. This process mitigates the negative impact of intervention delays, ensuring the agent learns to imitate expert policy faithfully.

W3: faithfulness clarification

The notion "faithfulness" is mainly about imitating human experts only, while still leveraging trajectory data from both human experts and the learning policy for data efficiency. This is to alleviate the negative impact of the suboptimial of learning policy trajectory data, and achieved by a faithful imitation objective that optimizes the difference between the behavior policy and the expert policy (see Section 4.1) with a discrimilator.

We clarify that our target is to imitate a hybrid dataset consisting of human expert trajectories and the learning policy trajectories for data efficiency. This hybrid dataset brings two key callenges: 1) changing distribution and 2) deviation trajectories caused by expert reaction delay. The changing distribution is addressed by formulating the whole imitation learning problem as dynamic regret, and the effect of deviation trajectories addressed by our proposed faithful imitation objective. This faithful imitation objective is used as a loss in dynamic regret.

Furthermore, we argue that dynamic regret promotes more faithful imitation compared to static regret. Our approach allows the model to track the most recent and highest-quality policy trajectories, which progressively converge toward the expert's policy. This avoids the bias from outdated, suboptimal data that would dilute a static regret objective, thereby leading to a more faithful imitation.

W4: Theoretical Analysis

Thanks for the comments, we will provide the following theoretical analysis on regret guarantees. Our theoretical analysis shows that our algorithm is no-regret and can eventually converge to the optimal policy.

(We encountered a rendering issue with long equations in the OpenReview rebuttal system. To address this, we have broken them down into smaller terms, which required introducing new notations (e.g., $k = i-1$) and may harm the paper's readability.)

We aim to prove that our FaithDaIL is no-regret and can eventually converge to the optimal policy, i.e.,

$\frac{1}{M} \mathbb{E}[\mathcal{R}_D] \to 0$

We first give some assumption:

• Bounded Parameter Space: The policy parameter space is effectively bounded due to regularization and optimizer constraints in practice.
• Lipschitz Loss: The imitation loss $\ell_i(\pi)$ is Lipschitz in $\theta_\pi$, assuming (1) $\pi(a|s)$ is Lipschitz, (2) importance weights $\omega_i^\star(s,a)$ are bounded, and (3) $\pi(a|s)$ is bounded away from zero.

Following the analysis from FTPL-D+, the expected dynamic regret of FaithDaIL satisfies:

$\mathbb{E}[\mathcal{R}_D] \le \mathcal{O}\left( M^{\frac{2}{3}} (V_M + 1)^{\frac{1}{3}} \right),$

where:

• $M$ is the number of rounds,
• $V_M = \sum_{i=2}^{M} \max_{\pi \in \mathcal{K}} |l(\pi, \mathcal{D}_i^B, \mathcal{D}_i^H) - l(\pi,\mathcal{D}_k^B, \mathcal{D}_k^H)|$, $k=i-1$ is the total variation of the loss sequence.

To establish that the dynamic regret is sublinear, it suffices to show that $V_M$ grows sublinearly with $M$. We formalize the implicit intervention rate $\beta_i$ and state the core assumption underlying its use:

• Definition (Implicit Intervention Rate $\beta_i$) :

$\beta_i = \mathbb{E}_{\pi^N_i } \left[ \frac{|\mathcal{D}_i^{H,\text{new}}|}{|\mathcal{D}_i^{H,\text{new}}| + |\mathcal{D}_i^N|} \right]$

• Assumption (Intervention Rate Decay) : This reflects the intuition that as the agent improves, fewer human interventions are needed over time.

$\lim_{i \to \infty} \beta_i = 0$

To relate $\beta_i$ to the trajectory distribution shift, we invoke the following result (adapted from DAgger, Lemma 4.1):

• Lemma (Distribution Divergence Bound): With $T$ as the MDP horizon,

$\|d_i^B - d_i^N\|_1 \le 2T\beta_i$

We now bound the single-step loss variation $\Delta_i = \max_{\pi} | \ell_i(\pi) - \ell_{i-1}(\pi) |$.

$|\ell_i(\pi) - \ell_{i-1}(\pi)| \leq |A - B| + |B - C|$

where $k=i-1$ :

$A = \mathbb{E}_{(s,a) \sim d_i^B}[w_i^\star(s,a) \log \pi(a|s)]$

$B = \mathbb{E}_{(s,a) \sim d_k^B}[w_i^\star(s,a) \log \pi(a|s)]$

$C = \mathbb{E}_{(s,a) \sim d_k^B}[w_k^\star(s,a) \log \pi(a|s)]$

1. Term 1 (due to distribution shift):：

$\left| A-B \right| = \left| \int_{(s,a)\in\mathcal{S}\times\mathcal{A}}[d_{i}^B(s,a) - d_{k}^B(s,a)] w_i^\star(s,a) \log \pi(a|s) ds da \right|$

$\leq N_1 \cdot N_2 \cdot \|d_i^B - d_k^B\|_1$

where:

$N1 = \|w_i^\star \|_{\infty}$

$N2 = \|\log \pi\|_{\infty}$

2. Term 2 (due to weight shift):：

$\left| B-C \right| = \left| \int_{(s,a)\in\mathcal{S}\times\mathcal{A}}[w_{i}^\star(s,a) - w_{k}^\star(s,a)] d_k^B(s,a) \log \pi(a|s) ds da \right|$

$\leq N_2 \cdot N_3 \cdot \|w_{i}^\star - w_{k}^\star\|_1$

where:

$N2 = \|\log \pi\|_{\infty}$

$N3 = \|d_k^B \|_{\infty}$

Step 1: Bounding distribution shift: Using triangle inequality and Lemma (Distribution Divergence Bound):

$\|d_i^B - d_k^B\|_1 \le 2T(\beta_i + \beta_k) + \|d_i^N - d_k^N\|_1$

Assuming the novice policy changes smoothly:

$\sum_{i=2}^M \|d_i^N - d_{i-1}^N\|_1 = o(M)$

and $\beta_i \to 0$, we have:

$\sum_{i=2}^M \|d_i^B - d_{i-1}^B\|_1 = o(M)$

Step 2: Bounding weight shift: Assuming that the variation in importance weights is primarily induced by the shift in behavior distributions, and that $V_i^\star$ evolves smoothly under $d_i^B$, we have:

$\|w_i^\star - w_{i-1}^\star\|_1$

$\le L_w \cdot \|d_i^B - d_{i-1}^B\|_1 + o(1)$

Thus:

$\sum_{i=2}^M \|w_i^\star - w_{i-1}^\star\|_1 = o(M)$

Then under the earlier Lipschitz assumptions, we assume relevant terms of N1, N2, N3 are uniformly bounded.

Conclusion: Sublinear functional variation.

$V_M = \sum_{i=2}^M | \ell_i(\pi) - \ell_{i-1}(\pi) | = o(M)$

Final Result: No-regret Guarantee

From FTPL-D+ analysis:

$\mathbb{E}[\mathcal{R}_D] = \mathcal{O}(M^{2/3}(V_M + 1)^{1/3}) = o(M), \quad \frac{1}{M} \mathbb{E}[\mathcal{R}_D] \to 0$

Hence, FaithDaIL is a no-regret algorithm in non-convex dynamic environments, with the learned policy converging to the round-wise optimal.

We thank the reviewer for the positive feedback on our responses and for raising this important follow-up question. Regarding this question:

We clarify that our target policy is the fixed expert policy. Specifically, we proposed a faithful imitation objective that directly sets the expert policy as the imitation target, while still leveraging mixed novice and expert data. Nevertheless, this objective can still be optimized within the dynamic regret framework, as clarified in our earlier response.

As we optimize for the expert policy, minimizing the dynamic regret ensures that the learner converges to the optimal policy—a process that does not inherently depend on the frequency of human intervention. Our empirical results suggest that convergence can be achieved under relatively sparse interventions, which highlights the practical robustness of our approach.

The reviewer's concern is exactly the limitation of prior HIL methods (IWR, HACO, and PVP), in which, the imitation objective is implicitly defined on the mixture of the expert and learner policies. And we believe quantifying human intervention frequency will be a good way to measure how good their solutions are theoretically.

We sincerely thank the reviewer for the thoughtful follow-up. We clarify that our method can approach the optimal expert policy conditioned on the learning-from-human-intervention setting. In this framework, a human expert ensures safety and alignment with human intent. Specifically, whenever the agent makes an error or deviates from human preference, the expert intervenes to guide it onto a correct trajectory.

This establishes a direct relationship between the learner's performance and the rate of intervention. A higher frequency of errors, which signifies a larger gap between the learner's policy and the optimal policy, necessitates more frequent interventions. Conversely, a significantly sparse intervention rate indicates that the learner's policy is successfully converging toward the expert's.

Here, we provide the theoretical justifications of the relationship between the round specific gap to the ground-truth optimal policy $\ell_i(\pi_i^N) - \ell_i(\pi^H)$ and intervention rate $\beta_i$ as follows:

(We encountered a rendering issue with long equations in the OpenReview rebuttal system. To address this, we have broken them down into smaller terms, which required introducing new notations and may harm the paper's readability.)

First, according to Eq. (8) in the paper, we have:

$\ell_i(\pi_i^N) - \ell_i(\pi^H) = \mathbb{E}_{(s,a) \sim d_i^B} \left[ \omega_i^*(s,a) \log \frac{\pi^H(a|s)}{\pi_i^N(a|s)} \right]$

According to the lemma in the preceding proof (see our response in W4), we have:

$\| d_i^B - d_i^N \|_1 \leq 2T \beta_i$

Let $f(s, a) = \omega_i^*(s, a) \log \frac{\pi^H(a|s)}{\pi_i^N(a|s)}$. It is known that the difference in expectations under two distributions can be bounded by their total variation distance:

$|A- B| \leq \sup |f| \cdot \| d_i^B - d_i^N \|_1$

where,

$A=\mathbb{E}_{d_i^B}[f]$

$B=\mathbb{E}_{d_i^N}[f]$

Combining the above inequalities, we obtain:

$\ell_i(\pi_i^N) - \ell_i(\pi^H) = \mathbb{E}_{d_i^B}[f]$

$\leq \mathbb{E}_{d_i^N}[f] + C \cdot \beta_i$

where $C = 2T \cdot \sup |f|$ is a constant.

The first term $\mathbb{E}_{d_i^N}[f]$ in the above equation measures the discrepancy between the novice policy and the expert policy during the novice's own exploration.

A larger $\mathbb{E}_{d_i^N}[f]$ indicates a greater deviation from human expectations, leading to a higher intervention rate $\beta_i$ since human experts intervene whenever the novice policy makes mistakes.

Next, we derive the relationship between $\mathbb{E}_{d_i^N}[f]$ and intervention rate $\beta_i$.

Since intervention is determined by human subjective judgment, we can assume that for a particular human expert, there exists a psychological threshold for intervention, beyond which the expert chooses to intervene. Formally, we define intervention event as follows:

When the divergence between the novice policy and the expert policy at a certain state $s$ exceeds a threshold $\delta$, a human intervention occurs:

$E_{\text{int}}(s) \equiv \\{ D_{KL}(\pi^H(\cdot \mid s) \,\|\, \pi_i^N(\cdot \mid s)) > \delta \\}$

We then relate the intervention rate $\beta_i$ to the event $E_{\text{int}}$:

$\beta_i = \mathbb{E}_{s \sim d_i^N(s)}$ I1 $

$

where $I1 = \mathbb{I}\left( E_{\text{int}}(s) \right)$

The policy gap expectation can be decomposed as:

$\mathbb{E}_{d_i^N}[f] = A2 + B2$

$A2 = \mathbb{E}_{s \sim d_i^N(s)}$ I1 \cdot f_s $

$

$B2 = \mathbb{E}_{s \sim d_i^N(s)}$ I2 \cdot f_s $

$

$I1 = \mathbb{I}\left( E_{\text{int}}(s) \right)$

$I2 = \mathbb{I}(\neg E_{\text{int}}(s))$

where

$f_s = \mathbb{E}_{a \sim \pi_i^N(\cdot \mid s)} \left[ \omega_i^*(s, a) \log \frac{\pi^H(a \mid s)}{\pi_i^N(a \mid s)} \right]$

is the policy gap at state $s$.

We provide upper bounds for both components:

For states likely to trigger intervention, we assume an upper bound of $f_s$ under these states as $C_{\text{high}}$:

$A2 \leq \beta_i \cdot C_{\text{high}}$

For states unlikely to trigger intervention, we assume a small constant upper bound of $f_s$ under these states as $\epsilon_{\text{safe}}$:

$B2 \leq \epsilon_{\text{safe}} \cdot (1 - \beta_i)$

Therefore, the term $\mathbb{E}_{d_i^N}[f]$ can be bounded as

$\mathbb{E}_{d_i^N}[f]$

$\le \beta_i \cdot C_{high} + (1-\beta_i) \cdot \epsilon_{safe}$

$= (C_{high}-\epsilon_{safe}) \cdot \beta_i+\epsilon_{safe}$

When the novice policy deviates significantly from the human expert policy, we can have $C_{high} > \epsilon_{safe}$. As the novice policy approaches the expert policy, $\epsilon_{safe}$ approaches 0.

In summary, we have:

$\ell_i(\pi_i^N) - \ell_i(\pi^H) \leq (C+C_{high}-\epsilon_{safe}) \cdot \beta_i+\epsilon_{safe}$

That is, the loss gap between the temporary optimal policy and the the ground-truth optimal policy at each round is bounded by the intervention rate $\beta_i$. As $\beta_i \to 0$, the learner will converge to the human expert policy.

Thanks for the reply. While I don't quite understand why the proposed approach can approach the optimal expert policy without any conditions. The optimization objective (Eq. 1 in the paper), i.e. the dynamic regret, compares the performance of the learned policy to the optimal policy up to the current round. Assume that human intervention is significantly sparse, then these temporary optimal policies can have a large gap to the ground-truth optimal policy. I would like to know if any further theoretical justifications can be provided.