Thank you for your review. We hope our response might provide sufficient evidence that our method is general enough to consider upgrading your score. In particular please note the additional experiments on a new environment, and the three additional variants (two already included in Appendix F, one new in the global rebuttal).

Q1. From theory to practical settings

A1. Please see our global review A3.

Q2. More environments

A2. We have included a set of Atari results under the global rebuttal. Please let us know any specific additional environments you'd like to see.

Q3. Latest GAIL variants

A3. While the results in the paper's main body indeed focus on GAIL and GAIL-DAC, Appendix F Table 2 already includes comparisons with two other recent GAIL variants -- BC-GAIL and WAIL. We will make these results more prominent in the next version of the paper. In addition, we have included a diffusion variant in the global rebuttal. Please let us know any specific additional variants you'd like to see.

Q4. Differences between C-GAIL and similar methods in GANs?

A4. Compared with GANs, the policy generator of GAIL involves an MDP transition, which results in a much more complicated dynamical system induced by a policy acting in an MDP rather than a static data generating distribution. Prior theoretical analysis and controllers are therefore inapplicable. We adopt different analysis (different ODEs) and controlling techniques (local linearization), to present a new stability guarantee, controller, and theoretical results for the different dynamical system of GAIL.

Q5. Exploration for entropy term

A5. Thank you for this comment. We understand that inclusion of the entropy term is critical to allow the generator policy to explore the state space. C-GAIL maintains this entropy term, while additionally adding a controller to stabilize the system. Removing the entropy term directly as you suggest is an interesting proposal, but we expect this would cause the algorithm to collapse to some local minimum without fully exploring the space.

Q6. Theoretical guarantee for other GAIL variants

A6. Our work presents a methodology to analyse the training process as a dynamical system and design a controller to stabilize its training. To design a controller for other GAIL variants, we would start with their objective functions and take a derivative to get their dynamical system respectively. The only difference is on the design of the generator's controller to ensure local linearization theorem. The controller for the discriminator will remain the same.

In this manner, since our implementation of C-GAIL only adds controllers for the discriminator, our C-GAIL is compatible with WAIL, LSGAIL and f-GAIL. We have provided our result compared with WAIL in appendix F.

Q7. Length of expert demonstration and how obtained

A7. Our main experiments follow the same settings as GAIL-DAC [1], which in turn followed vanilla GAIL [2] which used 50 timesteps per trajectory. The expert policies were trained with TRPO.

Q8. Drop in Hopper Figure 2, number of random seeds

A8. This is a good observation. We are not certain why there is a sudden dip in Wasserstein distance for Hopper. It's possible that, because the distance is computed between a low number of expert and generated trajectories, there is noise in our measurement, leading to these unexpected spikes.

Figure 2 is computed by averaging five random seeds. We apologize for the confusion as we did not originally include the error bars. We present the standard deviation as table in global rebuttal Q4 and will modify our figure.

Q9. Undefined notations

A9. Thank you for pointing it out. We will properly define them to avoid confusion. They are substitutes notation for $s$ and $a$ since we have the initial condition $s_0 = s$ and $a_0 = a$.

Q10. Table 2 missing expert return

A10. Thank you for noticing this. We will add an extra row for expert return. Since we are conducting on the same Mujuco tasks as Figure 1 and Table 1, the expert return is the same.

[1] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925, 2018.

[2] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565–4573, 2016.

Thank you for your review. We're pleased you have seen value in our new control-theoretic approach to understanding and stabilizing GAIL. We appreciate several nuanced observations you raised, and respond to these below.

Q1. One-step vs full GAIL

A1. Theoretically, our controllers work under one-step setting. We follow the insights from one-step setting and extend our controller to full GAIL as C-GAIL. It is difficult to know how to study the difference between the one-step and full setting analytically -- if able to do this, we could directly design controllers for the full setting! It is an interesting direction for future work. However, we have empirically studied the effectiveness of our C-GAIL under the full setting through our experiments. Also see our response A2 to R jmhJ, which notes similar approximations have been used in popular methods such as TRPO, and the global rebuttal A3.

Q2. Quantifying oscillations

A2. This is a good point. We have used the term "oscillations" informally in the text, without defining it rigorously. A direct way to interpret this is as variance in the reward curves across runs -- viewing the size of the shaded standard deviation in Figure 1 would be one way to quantify this (or reading off the final standard deviations in Table 2). Another way is to consider the oscillations between successive training time steps -- we do not have a single metric capturing this, but it can be viewed on Figure 1 also.

Q3. Design and implementation of the controllers

A3. Note that our controllers $u_1$ and $u_2$ only need to be derived once for a given dynamical system. Automating their design in an arbitrary new dynamical system is an interesting direction for future work, but beyond the scope of this paper. We have chosen to use additive controllers, which are implemented as an addition term to the objective functions (line 238-239). This is summarized in Algorithm 1. Notice that C-GAIL only introduces one additional hyperparameter $k$ (this is one of the attractions of our method). We provide an ablation study in appendix E, finding that C-GAIL is not particularly sensitive to the choice of $k$.

Q4. High initial value on C-GAIL-DAC

A4. Thank you for this interesting observation. To reiterate; Figure 2 presents the state Wassertein distance for GAIL-DAC and C-GAIL-DAC. The state Wassertein distance measures the difference between expert and generator distributions at a given point in training. Higher values indicate a larger difference between expert and generator. As noted by the reviewer, in 4/5 environments C-GAIL-DAC briefly has a higher (worse) metric than GAIL-DAC, though in the long run always ends up lower (better). We are not certain why this occurs, particularly as the reward curves are almost always strictly better for C-GAIL-DAC (Figure 1). Intuitively, since our controller for the discriminator encourages convergence to $\frac{1}{2}$, the corresponding generator may have a larger distance to the expert one initially. However, as the discriminator approaches its equilibrium, the generator may become stable too as a response.

Q5. Number of expert demonstrations

A5. Thank you for this feedback -- we will add a discussion on this in the next version of the paper. We used four expert demonstration to be consistent with the original GAIL-DAC paper [1]. As we understand, one of the advantages of GAIL is that it requires less expert trajectories than other methods (e.g. BC). It's possible that other environments and GAIL variants may require larger numbers of demonstrations (it also depends on the length of each trajectory). We do provide an ablation in Figure 3 with up to 18 demonstrations.

[1] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925, 2018.

Thank you for your review, we are pleased to have communicated the value of our control-theoretic approach. We hope our clarification around the biased-ness of GAIL, and addition of further experiments, might encourage an increase your evaluation of the paper.

Q1. The entropy term and bias

A1. We don't believe it's correct to say "the entropy term introduces subtle bias in but improves stability". The role of the entropy term is to allow the generator to explore the state-action space and discover how to reproduce the expert trajectories. But it does not improve stability directly. We have proved that the entropy term prevents the system from converging to the desired global equilibrium, it is not clear to us that there is some other "biased" equilibrium that it is pushed to instead -- empirical evidence comes from the oscillations in the GAIL-DAC training curves (Figure 1) suggesting there is no alternative more stable local equilibrium.

Q2. Ignoring differences on state distribution and performance

A2. We indeed ignore the effect of policy changes to the state distribution (switching $\rho_{\pi_E}$ with $\rho$) and only consider the influence of $\pi$ on actions in our one-step setting. We believe this is not uncommon in RL theory to simplify analysis. One example is TRPO [1], which similarly allows the original policy distribution to approximate the updated distribution and provides a theoretical justification for this approximation (Eq2 to Eq6 in its paper).

Notice that the methodology of starting with simplified setting in theory and extend it to general setting in practice is also commonly used in GANs (global rebuttal Q3).

Our intuition for why performance is still good with only a controller on the discriminator is that C-GAIL pushes the discriminator to its equilibrium at a faster speed by introducing a penalty controller centered at $\frac{1}{2}$. This leads GAIL’s training to converge faster with a smaller range of oscillation, and match the expert distribution more closely.

Q3. Biased regularizer and strong convexity

A3. We do not believe "bias" is the right term here. In control theory we aim for certain equilibria. As per our above response, it's not clear that GAIL with entropy converges to any equilibrium. Further more, we are not replacing this entropy term, but introducing an additional regularizer that does make the system stable around the desired equilibrium.

We are unsure what the convexity comment refers to. None of our results rely on this assumption.

Q4 More benchmarks

A4. Thank you for this suggestion. We have added additional experiments as per the global rebuttal.

Q5. Missing parenthesis

A5. Thank you for pointing it out. We will modify it.

Q6. One-step setting and lack of policy regularizer

A6. Please see our response A2 and global rebuttal A3.

[1] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. ICML, 2015.

Thank you for taking the time to review our paper. We are pleased we were able to communicate the strengths of the work effectively. We value your feedback on the gap between theory and practice. We have included a general response to this point in the global rebuttal, and here offer more targeted comments. We hope this, along with further experiments in game playing environments, and clarifying the importance of hyperparameter selection, might warrant an uplift in your score.

Q1 Theoretical controller to practice

A1. As noted by the reviewer, we were unable to implement the derived generator regularizer since we don't have access to the expert policy. We would like to emphasize that the theoretical contribution made by our work remains -- our theory is non-trivial, offering a new way to study GAIL, and insight into what an ideal generator regularizer would look like, which may inspire future work. We showed empirically that the discriminator alone is still of significant practical benefit by itself.

Q2. Continuous vs Discrete system

A2. Gradient flow [1] is a widely used technique across optimization theory to transform discrete gradient step updates to a continuous differential equation. This is critical to allow systems to be studied analytically. We agree with the reviewer that understanding what is lost in this conversion is of interest. However, given the ubiquity of the technique we feel it is beyond the scope of our work to study this in depth in our specific setting. We do note that our empirical results, which use discrete updates (algorithm 1), show the benefit of our continuous study does transfer to the discrete setting.

Q3. Hyperparameter assumption requirements

A3. Assumption 4.2 specifies an allowed range of hyperparameters in order for convergence to be guaranteed in theory. It is not clear how impactful this is when choosing $k$ at implementation time, since one needs to know $c$ ($p(s)$) for each state. So the allowed bounds are more valuable from a theoretical perspective.

Q4. Experiments on other tasks

A4. Thank you for this suggestion. We have added game environments (Atari) under the global rebuttal.

Q5 Selection of hyperparameters

A5. Thank you for this comment, which suggests we have not communicated the effect of the hyperparameter $k$ clearly. We provided an ablation study on $k$ in Figure 4. This shows that C-GAIL is effective under a wide range of $k$ values, from 0.1 to 10, with only marginal gains to be found by tuning it. As such, one of the benefits of our method is the insensitivity to $k$ and lack of need for tuning. We will emphasize this point more clearly in the next version.

[1] Conley, C., 1988. The gradient structure of a flow: I. Ergodic Theory Dynam. Systems, 8.

Thank you for your positive review. We are delighted that we were able to communicate the value of our work. We address your questions below, and have provided several new results in the global rebuttal.

Q1. Figure 2 standard deviation

A1. The results in Figure 2 are indeed aggregated from five random seeds. Thank you for pointing out that we did not report error bars on the plot. We present the standard deviation as tables in global rebuttal Q4 and will modify our figure.

Q2. Table 1 Hopper results exceed expert

A2. This is a good observation, and we do not have a comprehensive answer for why the agent average slightly surpasses expert performance. This also happened in the new DiffAIL results. It's possible that the agent overfits to one of the more successful demonstration trajectories. However, we notice that the expert return is within the error bars of our method.

Q3. Additional experiment domains

A3. We agree that our results could be more impactful with further domains. We have added in additional environment on Atari as per the global rebuttal.

Q4. Additional IL algorithms in related work

A4. Thank you for pointing out this gap in our related work, which we will add to the next version of our paper. Moreover, inspired by the reviewer's comment, we have conducted new experiments on DiffAIL in the global rebuttal.