Contractive Dynamical Imitation Policies for Efficient Out-of-Sample Recovery

We sincerely thank the reviewer for their valuable suggestions. Each comment has been addressed in the sequence provided, and we have incorporated additional improvements throughout the manuscript. An updated version of the manuscript is released.

1. Papers/references showcasing successful real-world deployments/applications on robotic systems

Learning dynamical systems has long-established literature, and many works have demonstrated real-world applications. Below are a few papers demonstrating successful deployment for real-world sensorimotor learning:

(1) Rana, Muhammad Asif, et al. "Euclideanizing flows: Diffeomorphic reduction for learning stable dynamical systems." Learning for Dynamics and Control. PMLR, 2020.

(2) Mohammadi, Hadi Beik, et al. "Neural contractive dynamical systems." The Twelfth International Conference on Learning Representations, 2024.

(3) Abyaneh, Amin, Mariana Sosa Guzmán, and Hsiu-Chin Lin. "Globally Stable Neural Imitation Policies." In IEEE International Conference on Robotics and Automation (ICRA), 2024.

2. Realizability of the trajectory generated by the proposed methods

Assessing the feasibility of the induced trajectories is a major challenge in model-free imitation learning. In general, imitation learning methods cannot fully address trajectory feasibility for out-of-distribution regions of the state space without additional mechanisms. Best mitigations typically involve data augmentation or expert-in-the-loop setups, as discussed in [1, 2]. In our case, while our method guarantees contractive recovery from out-of-distribution states, the feasibility of the generated trajectories cannot be ensured without access to additional data or a dynamics model of the robot. However, both these requirements conflict with the assumptions of our model-free and offline approach. In many practical tasks, trajectory feasibility is mitigated by passing the next state to a low-level controller, such as a model predictive controller, which generates collision-free torque or velocity commands for the robot. While this addresses the feasibility problem at the lower level, the limitation could persist at the planning level.

3. Setting the initial states in regions with a low state visitation density

We would like to highlight that Figure 10 in the original submission analyzes the impact of the spread of initial conditions using a task from the LASA dataset. For further details, please refer to our answer 9 to reviewer AXT4.

4. Could you try training a diffusion model based policy (with treating the problem as future state generation conditioned on current state and train it with a data augmentation that adds the same level of ellipse noise to conditioned input state) and evaluating it in your experiments? I suspect this would already generate a decent performance on the experiments you conducted.

While the representational power of diffusion-based policies has been demonstrated in recent work [3], these methods typically treat the diffusion process as a black box to model action distributions. As the reviewer points out, training on augmented data might yield desirable outcomes, but there are no theoretical guarantees for samples outside the augmented dataset. This highlights a key difference between black-box learning approaches, like diffusion policies, and our method, which is based on dynamical systems [4] with theoretical stability guarantees. In our approach, contractivity is ensured without requiring augmented data, even for out-of-sample initial states far from the training ones. Additionally, we note that diffusion-based policies generally rely on significantly more parameters (approximately 6.5e+07 [3]) and a larger number of demonstrations (typically 50–60). In contrast, our method employs a much simpler model with a maximum of 1.5e+02 parameters and is effective even with a single expert demonstration, as demonstrated in the paper, and is still capable of guaranteeing out-of-sample recovery for the entire state space.

References

[1] Brown, Daniel, et al. "Safe imitation learning via fast Bayesian reward inference from preferences." International Conference on Machine Learning. PMLR, 2020.

[2] Laskey, Michael, et al. "DART: Noise injection for robust imitation learning." Conference on robot learning. PMLR, 2017.

[3] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research, 2023.

[4] Ravichandar, Harish, et al. "Recent advances in robot learning from demonstration." Annual review of control, robotics, and autonomous systems 3.1 (2020): 297-330.

7. Approaching a goal from very directions would only be possible with loose contraction rates, which would defeat the main motivation of the paper.

We respectfully disagree with the assertion that multi-modal initial state distributions are incompatible with contractivity. Our experiments, such as those shown in Figures 4 and 5, successfully demonstrate the ability of our method to imitate multi-modal demonstrations.

Contractivity ensures that all trajectories converge to the target state from any initial condition, regardless of the contraction rate. In our experiments, trajectories starting from initial conditions within each mode contract together before ultimately converging to the target. This behavior highlights the desired transient dynamics alongside the steady-state guarantees, fulfilling the goals of the paper in both aspects.

8. Table 1 highlights the best mean MSEs, although the confidence intervals are quite large, which is a bit misleading.

While the confidence intervals are large in some cases, they are not misleading in selecting the best method. This is because our method achieves the lowest out-of-sample error, which is the most critical criterion for determining the best approach, both in terms of mean and variance. For in-sample errors, there are three cases where the baseline with the lowest mean does not have the lowest variance. To address this, in the final version of the paper, we will mark baselines that are not statistically significantly worse using Welch’s t-test.

The larger confidence intervals in Table 1 arise from the inclusion of multiple tasks with varying levels of difficulty within each dataset. For instance, in the Robomimic dataset, the 'lift' task is relatively easier to imitate than the 'transport' task, leading to larger confidence intervals.

9. Evaluating different methods for more distant initial states and states that appeared during the end of a demonstration

We highlight that Figure 10 in the original submission analyzes the impact of the spread of initial conditions using a task from the LASA dataset. In the most extreme cases, some initial states are close to the end of the demonstrations. As shown in the figure, SCDS demonstrates effective out-of-sample recovery, even when the initial states are far from the in-sample ones. To increase the visibility of this analysis, we referred to this ablation study at the beginning of Section 5 in the updated paper.

Following your and reviewer hPP6’s comments, we have also included Table 3 in the updated version, comparing SCDS with other baselines when starting from more distant initial states. The results, shown below, indicate that our approach performs significantly better in these scenarios, which is expected given our strong convergence guarantees.

10. Oftentimes, we are also interested in capturing the variance of the demonstration, which can be important for variable stiffness control. In that respect, the baseline SDS-EF yields arguably a better imitation.

Our method consistently outperforms SDS-EF in out-of-sample performance across all datasets and achieves comparable or superior in-sample results in our experiments. Notably, we evaluated our method on datasets similar to those used in the SDS-EF paper, including the LASA dataset and the Franka arm. Our experiments also include multi-modal cases, demonstrating that our approach can capture the variance in demonstrations. While SDS-EF might perform better in specific scenarios, such as variable stiffness control, this would represent a different problem than the one analyzed in our paper. Additionally, SDS-EF does not claim to specifically address such scenarios. Hence, it is not possible to conclude that SDS-EF yields arguably better results.

11. Were the metrics in Table 1 computed by weighting the demonstrations using Equation 9? Wouldn't this bias the results in favor of the proposed method?

The weighting method does not bias the results, as explained below. For training our approach and evaluating in-sample rollouts, the weighted loss in Equation (8) reduces to an average loss, as noted in Lines 304–306 of the submitted paper. This is the same loss function used in other methods. For evaluating out-of-sample rollouts, the weights in Equation (9) are applied consistently across our method and the baselines. Therefore, the metrics in Table 1 are computed consistently for all methods, ensuring no bias in favor of our proposed approach. Additional clarification about the weighting is provided in our response to reviewer cFmX."

12. Why isn't the contractivity in conflict with multimodal trajectory distributions?

The key factor enabling this behavior is the initial distance between trajectories, $\mathbf{z}_0^a - \mathbf{z}_0^b$, appearing on the right-hand side of the contractivity definition in Equation (2).

Qualitatively, if two trajectories start from nearby initial states, the upper bound on their distance (RHS of Equation (2)) decreases below a specific small threshold $\epsilon$ very quickly, causing them to converge towards the target point together. In contrast, if the initial states are farther apart, it takes longer for the upper bound on their distance to fall below the same $\epsilon$, allowing the trajectories to approach the target from different directions before eventually converging.

Importantly, the generated trajectories from a contractive DS converge to a unique point. This requires a common endpoint for all modes in a multimodal demonstration.

References

[1] Khansari-Zadeh, S. Mohammad, and Aude Billard. "Learning stable nonlinear dynamical systems with Gaussian mixture models." IEEE Transactions on Robotics 27.5 (2011): 943-957.

[2] Mirchevska, Branka, et al. "High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning." 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2018.

[3] Bahl, Shikhar, et al. "Neural dynamic policies for end-to-end sensorimotor learning." Advances in Neural Information Processing Systems 33 (2020): 5058-5069.

We are grateful for the reviewer’s constructive and detailed feedback. We have carefully addressed all the comments in the order they were raised and made further refinements to the paper. An updated version of the manuscript is released.

1. No account for the robot dynamics and therefore the feasibility of the planned motions.

Our method operates under the “model-free” assumption. For explanation, please refer to the answer to the second question of reviewer hPP6.

2. All demonstrations end in the same state, which is rarely the case for practical problems.

We have removed the assumption of a fixed final state in the updated version of our paper, as it is unnecessary. A contractive DS converges to a unique target state from any initial condition. While we imposed this characteristic on the data in the original submission, our framework can still be applied to data that does not adhere to this property. In such cases, the model learns an average behavior, converging to a single point rather than replicating each endpoint exactly. An example of this scenario, with demonstrations ending at different points, is illustrated in Figure 13 of the paper, demonstrating that this assumption was not required.

3. By applying DTW, the method is not able to imitate velocity profiles.

Our method can imitate any kinematic measurement of the expert’s behavior, including velocity, regardless of the loss function used. This is achieved by augmenting the state to include both position and velocity, hence, doubling the state dimensions. In Section 5.2, we demonstrated this capability on the LASA-4D and Robomimic-14D datasets, which include both the robot’s joint position and velocity. The results, reported in Table 1, show performance using both MSE and DTW loss functions. Thus, our approach is not limited to imitating position profiles but also learns and reproduces velocity profiles accurately.

4. The weighting for the MLE loss (Equation 9) is not derived from first-order principles.

While we acknowledge that our weighting method is heuristic, we believe it is sensible as it has the following properties. The weighted loss simplifies to the standard average loss for in-sample initial states, as explained in Lines 304-306 of the submitted paper and our answer 5 to reviewer cFmX. Hence, when using the weights in Equation (9), the weighted loss generalizes the average loss.

It provides an intuitive solution for out-of-sample initial states. For initial states different from those in the dataset, it is unclear whether the generated trajectory should be compared to which demonstration. However, it is reasonable to use a weighted combination of the losses with respect to each demonstration, where a higher weight is assigned to the demonstration with a closer initial state. An example depicting this scenario is provided in Figure 3. To our knowledge, previous imitation learning methods have not considered computing out-of-sample loss.

We hope this clarifies the rationale behind our approach.

5. Assumption 4.1 seems to be overly strong, as it assumes that all (rather than any) demonstrated initial states are close to $\hat{y}_0$.

Assumption 4.1 is indeed milder than your statement. For example, in the case of two expert demonstrations, Assumption 4.1 requires that the sum of distances from $\hat{y}_0$ to the two demonstrated initial conditions is smaller than $R$. This places $\hat{y}_0$ inside an ellipse with focal points at the two demonstrated initial conditions—a much larger region than requiring all demonstrated initial states to be close to $\hat{y}_0$, as you noted, which corresponds to the intersection of two circles.

We identified and corrected a small typo in Assumption 4.1 in the updated paper. This correction does not affect the proof or the subsequent theorem.

6. The generated trajectory might not start at the correct initial state, due to using the pseudo-inverse.

This is correct. When $P_{\boldsymbol{\theta}}$ has full column rank, the pseudoinverse coincides with the left inverse, resulting in $\hat{y}(0) =y_0$, and the generated trajectory starts at the correct initial state. Otherwise, it provides the least-squares approximation to $y_0$.

Based on your comment and comments from other reviewers, we have added this clarification after Equation (6) in the updated paper to highlight the distinction between $\hat{y}(0)$ and $y_0$.

Assumption 4.1 is indeed milder than your statement. For example, in the case of two expert demonstrations, Assumption 4.1 requires that the sum of distances from $\hat{y}_o$ to the two demonstrated initial conditions is smaller than $R$. $\hat{y}_o$ inside an ellipse with focal points at the two demonstrated initial conditions—a much larger region than requiring all demonstrated initial states to be close to $\hat{y}_o$, as you noted, which corresponds to the intersection of two circles.

I don't understand. Consider a 2d-coordinate frame with demonstrations at [-0.5,0] and [0.5,0] and assume $R=1$. The sum of the distances at the position [-1,0], which is on the ellipse and $R$-close to one of the demonstrations is $0.5+1.5>R$. More generally, the sum of distances must always be larger than the maximum distance, and therefore we would need to be close to all demonstrations.

Qualitatively, if two trajectories start from nearby initial states, the upper bound on their distance (RHS of Equation (2)) decreases below a specific small threshold very quickly, causing them to converge towards the target point together. In contrast, if the initial states are farther apart, it takes longer for the upper bound on their distance to fall below the same $\epsilon$, allowing the trajectories to approach the target from different directions before eventually converging. Importantly, the generated trajectories from a contractive DS converge to a unique point. This requires a common endpoint for all modes in a multimodal demonstration.

But how about two trajectories that start at the same point and split mid-trajectory? E.g. moving around an obstacle from both sides?

Thank you for taking the time to go through our responses. We address your follow-up questions below. Please let us know if we can clarify further and improve your opinion of our work.

Consider a 2d-coordinate frame with demonstrations at [-0.5,0] and [0.5,0] and assume $R=1$. The sum of the distances at the position [-1,0], which is on the ellipse and $R$-close to one of the demonstrations is $0.5+1.5>R$. The sum of distances must always be larger than the maximum distance, and therefore we would need to be close to all demonstrations.

Assumption 4.1 does not state that $\hat{\mathbf{y}}_0$ must be $R$-close to one of the demonstrations. Instead, it requires that the sum of distances to all demonstrations is smaller than $R$, which defines an ellipse.

In your example with demonstrations at $[-0.5, 0]$ and $[0.5, 0]$, the point $[-1, 0]$ is not on the ellipse with $R = 1$. Instead, as you computed, it lies on the ellipse with $R = 2$ ($\vert-1-(-0.5)\vert + \vert-1-0.5\vert = 2 = R$).

To summarize, we consider the set of initial states where the sum of their distances to all demonstrations is smaller than a given $R$. In 2-dimensions with two demonstrations, this corresponds to an ellipse. As you noted, this implies that the distance to each demonstrated initial condition is smaller than $R$.

We believe Assumption 4.1 is not restrictive for two reasons:

• The ellipse can be made arbitrarily large by selecting a larger $R$, though this would loosen the upper bound on the loss. This trade-off is reasonable, as weaker performance guarantees are expected for initial points far from the demonstrations.

• Contractivity and convergence to the target are global properties that hold for all initial states, including those outside the ellipse. The ellipsoidal assumption is only employed for deriving performance bounds.

We hope this clarifies the reasoning behind Assumption 4.1 and addresses your concern.

How about two trajectories that start at the same point and split mid-trajectory? E.g. moving around an obstacle from both sides?

Since the expert is assumed optimal [1], having diverging situations like the one you described means that the expert behavior is stochastic. Our method focuses on deterministic policies, which are well-suited for deterministic demonstrations. To clarify this, we will explicitly state in the final version of the paper that the initial conditions of expert demonstrations are distinct. To the best of our knowledge, all dynamical system-based imitation learning approaches operate under this assumption (see for example [2, 3]).

References

[1] Ahmed Hussein et al. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 2017.

[2] Rana, Muhammad Asif, et al. "Euclideanizing flows: Diffeomorphic reduction for learning stable dynamical systems." Learning for Dynamics and Control (L4DC), 2020.

[3] Hadi Beik Mohammadi et al. Neural contractive dynamical systems. In The Twelfth International Conference on Learning Representations (ICLR), 2024.

Thank you for your thoughtful and detailed comments. We have responded to your feedback point by point, as originally outlined, and have also implemented further improvements to the paper. An updated version of the manuscript is released.

1. Misuse of the term "policy"

We agree that there is a slight misuse of the term 'policy.' In our paper, the policy mapping is split into two steps: the DS in Equation (4) maps the current state to a desired future state, and then a low-level controller calculates the action needed to reach this target state. This process, as illustrated in Figure 15, means that the DS and low-level controller together constitute the policy (there was a typo in Figure 15 in the original submission, please refer to the updated version).

To address this, we have added further clarifications in Section 2.3 of the updated paper explaining how the DS and low-level controller form the policy and highlighted that we refer to the DS alone as the 'policy' for simplicity.

2. Does the work learn an implementable controller/policy for the system?

Yes. Indeed, we implemented our method on the Franka arm and Jackal mobile robots using an off-the-shelf low-level controller in the Isaac Lab simulator. The only required step for implementing our controller is designing the low-level controller which maps the states planned by the DS in our policy to motor commands. As discussed in our initial response to reviewer AXT4, designing such controllers is a field of independent research. Furthermore, this reliance on a low-level controller is common in imitation learning [1] and reinforcement learning [2] methods that operate at the level of high-level actions (e.g., joint positions) rather than low-level (e.g., torque) commands. Thus, this reliance is not specific to our method. Additionally, the sim-to-real gap for controllers that output the joint position, such as ours, is significantly smaller than those outputting lower-level actions [3]. Therefore, we expect a smooth transition to real-world applications.

3. The left inverse does not exist in general.

We do not use the left inverse but rather the Moore–Penrose pseudoinverse, which exists for any matrix and provides a generalized solution in cases where a left inverse does not exist.

When $P_{\boldsymbol{\theta}}$ has full column rank, the pseudoinverse coincides with the left inverse, resulting in $\hat{y}(0) = y_0$. Otherwise, it provides the least-squares approximation to $y_0$ [4].

Based on your comment, we have added this clarification after Equation (6) in the updated paper to highlight the distinction between $\hat{y}(0)$ and $y_0$.

4. Equation (9) is the harmonic mean, not the geometric mean, as mentioned on page 16.

Page 16 applies the inequality of arithmetic and geometric means (AM-GM inequality) to establish a relationship, but it does not state that Equation (9) is a geometric mean. Based on your feedback, we have extended the proof of Theorem 4.1 in the updated paper to clarify how the AM-GM inequality is employed.

5. Where the submitted experiment code uses the weights $\lambda_m(\hat{y}_0)$?

We clarify how the weights $\lambda_m(\hat{y}_0)$ are used in training and evaluation stages:

• Training: The average loss is used during training. As you mentioned, the weights $\lambda_m(\hat{y}_0)$ are binary (0 or 1) when the initial state matches a dataset sample. Consequently, as noted in Lines 304–306 of the submitted paper, the weighted loss simplifies to an average loss. Using only the initial conditions in the dataset ensures fairness when comparing our method to other baselines, as it prevents our approach from utilizing additional initial condition samples.

• In-sample rollout evaluation (yellow rows, Table 1): In this case, the weights remain binary and the loss again reduces to an average loss.

• Out-of-Sample rollout evaluation (green rows, Table 1): For out-of-sample rollouts, where the initial conditions differ from those in the dataset, the weights are no longer binary and are adjusted accordingly. In this case, we use the weighted loss to evaluate our algorithm as well as the baselines. For further details on why the weights are necessary for out-of-sample rollouts, please refer to our response to reviewer AXT4.

References

[1] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research, 2023.

[2] Bahl, Shikhar, et al. "Neural dynamic policies for end-to-end sensorimotor learning." Advances in Neural Information Processing Systems, 2020.

[3] Salvato, Erica, et al. "Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning." IEEE Access 9 (2021): 153171-153187.

[4] Penrose, E. T. (1956). Towards a theory of industrial concentration.

We appreciate the reviewer’s insightful feedback and have addressed each comment in the order it was presented. Furthermore, we made additional refinements to enhance the paper’s quality according to the comments. An updated version of the manuscript is released.

1. Computational time was not provided.

The computational times of our method and other baselines for different datasets are detailed in Appendix E6. To increase visibility, we have now included a direct reference to these results at the beginning of Section 5 in the main body of the updated paper. Our results indicate that, when training all baselines for the same number of epochs, our method is less computationally efficient in lower-dimensional datasets but is comparable in high-dimensional ones. However, this comparison is conservative, as our approach typically requires fewer epochs to achieve similar performance, resulting in a shorter actual training time. Importantly, our method offers a stronger contractivity guarantee, compared to only a stability guarantee in other baselines and no guarantees in naive behavioral cloning, which we believe offsets the time trade-off.

2. Using RENs and coupling layers increases the complexity. How does it compare with the baselines? While RENs and coupling layers increase the complexity of our policy representation, they bring two key advantages: (1) high flexibility and (2) guaranteed contractivity for any parameter choice, eliminating the need for contractivity constraints during training (see Lines 58–64).

A comparison with baselines using simpler policy representations is provided in Lines 65–74. These simpler baselines lack inherent contractivity and therefore require constrained optimization for training, which is computationally intensive. To mitigate this, they assume specific policy classes, which reduces expressiveness.

The only baseline that, like ours, inherently guarantees contractivity through its policy representation is the approach by [1]. However, as noted in Lines 75–81, their policy structure is not necessarily simpler than ours.

3. The generalization bounds must contain sample complexity results to reflect the sample requirements for the method.

Our bounds in Theorem 4.1 and Corollary 4.1.1 inherently reflect sample complexity through the parameter $M$, which denotes the number of expert demonstrations. For a fixed $R$, representing a fixed level of uncertainty in the initial conditions, the upper bounds in Theorem 4.1 and Corollary 4.1.1 become tighter as $M$ increases. This aligns with the expected behavior of improved generalization with more data.

In the extreme case where $M \rightarrow \infty$, the conservatism due to the uncertainty in the initial state (captured by term (ii) in Theorem 4.1 and Corollary 4.1.1) vanishes. This demonstrates that the bounds are non-vacuous.

4. Experiments on the Snake Environment employed in the SNDS paper.

We have tested our approach on the Messy Snake dataset following the same experimental setup as the SNDS paper and reported the results in Appendix E.7 of the updated version. As shown in Figure 15, our method (SCDS) generates contracting trajectories that better mimic the expert data compared to SNDS. Quantitatively, SCDS achieves an average DTW error of 0.074, significantly outperforming SNDS, which has an average DTW error of 0.158.

5. The paper assumes that there is a given final state.

We have removed the assumption of a fixed final state in the updated version of our paper, as it is not required. This assumption was inspired by the LASA dataset, in which all trajectories reach a given final state, but is not integral to our framework. A contractive DS converges to a unique target state from any initial condition. While we imposed this characteristic on the data in the original submission, our framework can still be applied to data that does not adhere to this property. In such cases, the model learns an average behavior, converging to a single point rather than replicating each endpoint exactly. An example of this scenario, with demonstrations ending at different points, is illustrated in Figure 13 of the paper, demonstrating that this assumption was not required.

6. Sensitivity to the quality of expert demonstration.

As is typical in imitation learning, our framework assumes that the expert demonstrations represent an optimal policy. This assumption is foundational in imitation learning, as discussed on page 6 of [5]: 'In imitation learning, the demonstrations provide the optimal action to a given state, and so the agent learns to reproduce this behavior in similar situations.' Addressing cases with suboptimal demonstrations is beyond the scope of this work.

7. Does the method interact with the environment during learning?

No, our method does not interact with the environment during learning. Instead, it relies on an offline dataset, as introduced in Equation (1) of the paper.

8. How critical is the sensitivity to the contraction rate when we have to run the algorithm on some new task?

Choosing the contraction rate is not a critical issue since it can be learned from data, as detailed in Appendix E2, allowing it to adapt to different tasks as needed. Our experiments in Figure 11 show the efficacy of learning the contraction rate.

References

[1] Hadi Beik Mohammadi, Søren Hauberg, Georgios Arvanitidis, Nadia Figueroa, Gerhard Neumann, and Leonel Rozo. Neural contractive dynamical systems. In The Twelfth International Conference on Learning Representations, 2024.

[2] Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 2020.

[3] Amin Abyaneh, Mariana Sosa Guzm ́an, and Hsiu-Chin Lin. Globally stable neural imitation policies. In IEEE International Conference on Robotics and Automation (ICRA), 2024.

[4] S Mohammad Khansari-Zadeh and Aude Billard. Learning stable nonlinear dynamical systems with Gaussian mixture models. IEEE Transactions on Robotics, 2011.

[5] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 2017.