EvIL: Evolution Strategies for Generalisable Imitation Learning

We sincerely appreciate the reviewer's insightful feedback and their thorough understanding of the relevant literature. We are particularly encouraged by the recognition of our approach's generality and wide applicability.

Addressing the reviewer's concerns:

• While we acknowledge that our current experiments do not exhibit strong scalability, it is important to note that IRL is typically evaluated on MuJoCo environments, which have a smaller observation space compared to some of the environments we employed. Within these environments, we achieved performance parity with the AIRL baseline.
• The papers mentioned have been added to the background under the Evolution Strategies subsection (2.1). While Niekum et al. (2010) do utilize evolution strategies to train a reward function, their objective is to enhance RL performance. In our approach, this is not feasible due to the absence of a reward signal. Similarly, Elfwing et al. (2018) also optimize for a reward function, but our framework operates under the assumption that no reward function is available. Evolved Policy Gradient (Houthooft et al., 2018) employs a similar setup with an ES outer loop optimizing for rewards in the inner loop. However, the optimization targets differ between the outer and inner loops in both approaches. To summarise, our approach is fundamentally different for all the papers above as we don’t assume access to a reward function signal we can optimise, but instead we optimise for a cross entropy loss against the expert’s trajectories.

Addressing the reviewer's specific questions:

• The right-side plots in Figure 1 illustrate how the reward function generalizes to the test environment during optimization on the training data. Notably, the reward function is never optimized on the test data or environment; rather, we solely evaluate its performance there.
• The original reward function that both AIRL and EvIL strive to recover remains the same. However, the reward function recovered by EvIL exhibits superior generalization to the test environment.
• The plot demonstrates that EvIL, when combined with some knowledge of the reward function (as is often the case in real-world problems), can enhance performance in the offline setting. This is shown in Figure 2a. In Figure 2b, we show the performance of Offline Evil, here the transition function is learned only through Evolution Strategies and a Cross Entropy loss between the trained agent and the expert trajectories. This suggests our ability to recover transition dynamics from the data. In Figure 2d, we show the performance of EvIL where the dynamics model is learnt through supervised loss. Figure 2f, where we show the performance of Online EvIL, is not directly comparable to the other plots, as the optimisation process has access to more information (i.e. the environment). Offline EvIL did not match the performance of the expert (and Online EvIL) as it was only trained on a single expert trajectory. We have additional results on the Reacher environment (with more expert data) that intuitively demonstrate improved performance as the number of expert trajectories increases. The results have been added to the Appendix.
• In the experiment depicted in Figure 4, we employ a partially known transition function and aim to uncover the remaining unknown variables. Our results demonstrate the successful recovery of these unknown variables, including obstacle position and size. Therefore it is the transition function, rather than the reward function, that serves as the source of this information.

We wholeheartedly appreciate the reviewer's encouraging feedback and their recognition of the intriguing application of our approach. We are delighted that the reviewer acknowledged the advantages of utilizing non-differentiable rewards and training methods.

Addressing the reviewer's concerns:

• We have addressed the reviewer's suggestions by including the BC hyperparameters in the appendix, defining the acronyms, and providing detailed descriptions of the environments.
• The transferability and generalizability of the recovered reward function is what we are testing for in the test version of our environment (e.g. different joint dampening in Reacher and a maze vs no-maze setting in the Gridworld)
• AIRL stands for Adversarial Inverse Reinforcement Learning, which is essentially equivalent to Generative Adversarial Inverse Reinforcement Learning (GAIL). In AIRL/GAIL, there are two players: a discriminator (used as reward function) and a player. The reward function player tries to pick out differences between the current learner policy and the expert demonstration, while the policy player attempts to maximise this reward function to move closer to expert behaviour. This setup is effectively a GAN. For a more theoretical explanation, please refer to [1].
• We are currently in the process of expanding our method to MuJoCo tasks by changing environment parameters like link lengths and joint stiffness to generate a diverse range of dynamics. This approach generally surpasses BC due to its foundation in Inverse Reinforcement Learning (training the agent in the inner loop) instead of simply imitating the expert's actions. BC has been shown to suffer from compounding errors and poor generalization.

To address the reviewer's questions:

• The Gridworld and Reacher environments were chosen because they pose a non-trivial challenge in recovering the reward function. The objective in these environments is to change position at the beginning of each episode, unlike other environments like CartPole which rely solely on a positive reward signal.

[1] Swamy, Gokul, et al. "Of moments and matching: A game-theoretic framework for closing the imitation gap." International Conference on Machine Learning. PMLR, 2021.

We express our gratitude to the reviewer for their valuable feedback and for recognizing the novel contributions of our work, particularly the L1 distillation technique and the dynamic adjustment of the number of inner loop gradient updates to prevent gradient vanishing issues.

Addressing the reviewer's concerns:

• While the number of inner loop steps for both AIRL and EvIL is the same, in the EvIL case, we leverage the power of parallel computation by training 128 agents simultaneously. This parallel training strategy significantly enhances the computational efficiency of our approach.
• To address the reviewer's request, a description of the AIRL implementation, along with the corresponding hyperparameters, has been included in the Appendix.

Question responses:

• The comparison with gradient approaches is the one we did by comparing with AIRL, the standard approach in this case

• Optimizing a potential-based shaping function using conventional meta-learning methods poses a significant challenge. As potential-based shaping only affects training speed, and not the optimal policy, it cannot be optimised with first-order methods. EvIL presents a perfect solution, as it is a zeroth-order optimisation method. The shaping objective could be optimized concurrently with the standard Cross-Entropy objective, resulting in better-shaped reward functions that are faster to optimize.

• Yes, we meant the dynamics models are trained, this has been fixed in the original paper

• The reward function for the Reacher environment is the standard one available in Gymnax and Brax, which utilizes a continuous reward signal based on the distance between the agent's end effector (“fingertip”) and the goal position. In the Gridworld environment, a reward of 1 is granted if all goals are discovered in the correct sequence before the episode ends; otherwise, the reward is 0. Similarly, in the goal point environment, the reward is only provided when the agent reaches the goal. This description has been added to the paper’s Appendix.

We sincerely appreciate the reviewer's comprehensive review of our work.

Addressing the reviewer's concerns:

• We are currently engaged in expanding our method's applicability to MuJoCo tasks, where we intend to introduce variations in environment parameters such as link lengths and joint stiffness to create a diverse range of dynamics.
• The iterative nature of our method, involving multiple rounds of training, aligns with standard practice for IRL methods. Both primal IRL approaches like GAIL/AIRL and dual IRL techniques like MaxEnt IRL are iterative procedures. Additionally, our method is not slower than standard IRL approaches, but is comparable due to the massive parallelization we are able to achieve via JAX. As an example, the PureJaxRL library archives a 1000x speed up over other standard PyTorch RL implementations.
• As you rightly pointed out, our method's strength lies in its ability to optimize non-differentiable reward functions through a non-differentiable optimization process. Moreover, we emphasize behavioral prediction outside the training distribution and demonstrate our method's superior performance compared to standard AIRL. While AIRL performs satisfactorily within the training environment, our experiments highlight our ability to achieve parity with traditional methods while expanding the range of applicable reward model families. For instance, we can optimize for symbolic expressions of environment features, as demonstrated in the recent EUREKA paper [1]. Additionally, we show better performance in the test environments, indicating rewards recovered by EvIL generalize better.

Question response:

• Yes, the expert trajectories are indeed sampled from a single environment.

[1] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models." arXiv preprint arXiv:2310.12931 (2023)

[2] Lu, Chris Xiaoxuan et al. “Discovered Policy Optimisation.” ArXiv abs/2210.05639 (2022)