PN-GAIL: Leveraging Non-optimal Information from Imperfect Demonstrations

Q1. Missing definition: in Section 3, what $\delta$ represents. Please define $\delta$ when it is first introduced in the paper.

A1. Thank you for pointing it out! $\delta$ represents Dirac delta function. We have added the above explanation to the revised version.

Q2. Needing clarification: PN-GAIL$\backslash$BSC: PN-GAIL without balanced semi-conf (BSC) classification--does this mean no classification used or SC used?

A2. We apologize for the inaccurate statement. PN-GAIL$\backslash$BSC stands for using SC classification. We have clarified the relevant explanations in the revised version.

Q3. Lack of analysis for experimental outcomes: it is necessary to provide more detailed explanations and discussions regarding those figures.

A3. Thank you for your suggestion. In Figure 3, the difference between the performance of PN-GAIL and PN-GAIL$\backslash$PN indicates that there is a preference in the imperfect demonstrations, resulting in the poor performance of the 2IWIL follow-up method. The performance gap between the performance of PN-GAIL and PN-GAIL$\backslash$BSC indicates that the prediction confidence of SC classification is not accurate enough, which affects the subsequent training. If the performance gap is not significant, it means that the above problems are not obvious or do not affect the final results. For example, in the Pendulum-v1 environment, we designed a high percentage of non-optimal demonstrations. In Figure 3, the performance of PN-GAIL and PN-GAIL$\backslash$BSC are close to and higher than that of PN-GAIL$\backslash$PN, indicating that there is little difference between the BSC classification and the SC classification, and there is a preference in imperfect demonstrations. This is also in line with the design of our dataset and the results of the confidence classifier. We have added the above explanation to the revised version.

Q4. To verify BSC outperform SC in 2IWIL, straightforward comparisons are PN-GAIL(BSC) vs. PN-GAIL(switch to SC) vs. 2IWIL(switch to BSC) vs. 2IWIL(SC). Why do you consider the way of comparisons that are presented in the paper?

A4. This is because a direct comparison of the difference between the classification results and the true value is more accurate, convincing, and quantifiable. We have compared the performance of PN-GAIL(BSC) with PN-GAIL$\backslash$BSC (which means using SC) in Figure 3. In addition, we have also added comparison between PN-GAIL$\backslash$PN (2IWIL with BSC) and 2IWIL (SC) in Figure 3. The results of both comparisons suggest that BSC is superior to SC.

Q1. Although $\alpha$ and $\beta$ are intended to play distinct roles in Theorem 4.2, they are selected to be identical in Algorithm 1, which may affect the overall applicability of the algorithm.

A1. We do not directly choose $\alpha$ and $\beta$ as the same value. In fact, after assuming that the covariances are small enough relative to the variances, the final approximations of $\alpha$ and $\beta$ happen to be the same. In addition, we empirically validate the appropriateness of our chosen values of $\alpha$ and $\beta$ in Common Response point 6. The results demonstrate the rationality of this setting.

Q2. Could the authors highlight expert performance more prominently in the figures to enhance clarity and interpretability?

A2. Thank you for your valuable suggestion. Following your suggestion, we have redrawn Figures 2, 3, 4 and added a blue dotted line to indicate the performance of the optimal policy.

Q3. Would varying values of $n_u$ and $n_c$ significantly impact the performance of the proposed method?

A3. We show the PN-GAIL performance at different $n_u$ and $n_c$ ratios in Figure 4 (a), Figure 4(b), and the PN-GAIL performance when $n_u$ is reduced in Appendix B.3 Figure 6. The results show that decreasing $n_u$ has a greater impact on method performance than changing the ratio of $n_u$ and $n_c$. In practice, when choosing the dataset size ($n_u+n_c$), it is better to ensure that the optimal demonstrations information contained in the dataset is sufficient to learn an optimal policy. When selecting the label ratio ($\frac{n_c}{n_c + n_u}$), we need to make the distribution of the label dataset as consistent as possible with the distribution of the whole dataset.

Q4. Provide more intuition of why GAIL and 2IWIL fail, but the proposed method succeeds. Why does GAIL predict 5.0 for some of these data points?

A4. Thank you for your valuable comments. We have redrawn Figure 1 and used gradient colors for different confidence scores. We explain Figure 1 with an example in Section 4.1, where the data points represent the equivalent confidence level at the time of training. Specifically, we consider the goals of GAIL, 2IWIL:

$\min_\theta \max_w E_{x\sim p_\theta}\left[\log D_w(x)\right]+ E_{x\sim p}\left[\frac{r(x)}{\eta}\log(1-D_w(x))\right].$

Expand the second term, which is $\sum p(x) \frac{r(x)}{\eta}\log(1-D_w(x))$, and since GAIL has no confidence scores, $\frac{r(x)}{\eta}$ is always equal to $1$. The coefficient for each state action pair $x$ in given demonstrations at the time of training is $p(x) \frac{r(x)}{\eta}$. Therefore, if there is a higher probability of $x_1$ appearing in imperfect demonstrations, such as $p(x_1) = 5p(x_{other})$. Then its coefficient is $p(x_1)\frac{r(x_1)}{\eta} = p(x_{other})\frac{5r(x_1)}{\eta}$, which equates to the confidence score of $x_1$, five times that of the original. For GAIL, since the confidence scores are all considered to be $1.0$, it is $1.0\times5=5.0$.

For PN-GAIL, by increasing the non-optimal information of the demonstrations, the higher the probability of a demonstration, the third term of Eq 11 will balance the influence of the second item due to the weight change, so that the score given by discriminator $D$ is more accurate. We have added the above explanations to the revised version.

Q5. To further contextualize this work within imitation learning (specifically, driver behavior imitation), it would be beneficial to incorporate additional relevant studies.

A5. In fact, we have already attempted the proposed algorithm in the field of autonomous driving and achieved preliminary results. Due to time and space limitations, we will publish such results in our future work.

Thanks to the authors for providing a detailed response. Based on the revisions and clarifications, I believe this paper meets the standards for acceptance. As such, I decide to maintain my score.

For the final camera-ready version, I encourage the authors to consider citing additional relevant references (e.g., the one we reviewers suggested [2][3][4][5] and some recently published works) to further strengthen the paper’s impact and completeness. While I totally understand that conducting additional experiments may be challenging at this stage, it would still be valuable to include comparisons and discussions in the related work section.

driver behavior imitation:

[2] Ruan, Kangrui, and Xuan Di. "Learning human driving behaviors with sequential causal imitation learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 4. 2022.

[3] Hawke, Jeffrey, et al. "Urban driving with conditional imitation learning." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.

Imperfect Demonstrations:

[4] Li, Ziniu, et al. "Imitation learning from imperfection: Theoretical justifications and algorithms." Advances in Neural Information Processing Systems 36 (2024).

[5] Yang, Hanlin, Chao Yu, and Siji Chen. "Hybrid policy optimization from Imperfect demonstrations." Advances in Neural Information Processing Systems 36 (2024).

Thank you for your valuable feedback and pointing out these relevant references. We have included these references and further strengthened the comparisons and discussions to related work and literature review in the revised paper.

Q1. The method's reliance on confidence scores may limit its applicability when it's difficult to assign confidence levels directly.

A1. Thank you for the insightful comments. Our method needs to label a small percentage of confidence scores, which can be obtained by averaging multiple expert labels (as long as trajectory preferences are obtainable). To further validate the superiority of confidence scores over simple trajectory preference ranking, in Common Response point 4, we compare the performance of PN-GAIL with CAIL, TREX, and DREX (typical trajectory preference-based methods). The results show that PN-GAIL outperforms other methods, achieving the highest rewards.

Q2. Related to my disagree on the claim that 'GAIL treats non-optimal demonstrations as if they were optimal' in the weaknesses section, please could you provide a counterargument?

A2. Thank you for this helpful comment. We have re-written this statement to avoid any potential confusion and also provided additional experimental results to further demonstrate this point. When expert demonstrations include non-optimal examples, GAIL aims to reproduce the overall distribution of these demonstrations, causing both optimal and non-optimal instances to be weighted equally. Consequently, if non-optimal demonstrations dominate, the learned policy will also reflect most non-optimal behaviors. In contrast, PN-GAIL mimics optimal demonstrations while minimizing the influence of non-optimal ones by assigning different weights to each type of demonstration and considering the associated positive and negative risks. This is also reflected in the poor performance of GAIL and the best performance of PN-GAIL in our additional experimental results in Appendix B.3 Figure 8.

Q3. Lacks comparison with other advanced IL algorithms, such as f-IRL.

A3. Following your constructive comments, we have demonstrated the performance of f-IRL [1] in Common Response point 5. However, since f-IRL is not designed for imperfect demonstrations, it does not work well because it reproduces all demonstrations with the same weight.

Q4. Theorem 1's bound relies on knowing the variances, how practically useful is the bound in scenarios where variance values are unknown or hard to assess?

A4. We are not sure if you are referring to Theorem 4.4 (If not, please let us know). For Theorem 4.4, the bound given is related to $\alpha$ and $\beta$. For computational convenience, we assume that these covariances are sufficiently small, and take a fixed approximation of $\alpha$ and $\beta$. The experiments in Common Response point 6 verifies the rationality of the approximation. When $\alpha$ and $\beta$ use a fixed approximation, the bound given is independent of variance.

Q5. Theorem 2's bound may become quite loose if the Rademacher complexity is high.

A4. Thank you for your insights. This phenomenon is indeed possible, and we believe that overfitting is one of the contributing factors. However, it can be mitigated through techniques such as regularization or dropout. Additionally, we can select different classification models based on the size of the dataset. For larger datasets, a more complex model may be appropriate, as the abundance of data helps reduce the risk of overfitting. Conversely, for smaller datasets, it is advisable to opt for a simpler model.

[1] Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pp. 529–551. PMLR, 2021.

Q1. Conducting experiments with more extreme demonstration ratios could more clearly demonstrate the scenarios in which PN-GAIL offers a distinct advantage over baseline methods.

A1. In the Pendulum-v1 environment, the ratio of the demonstrations is $\pi_{\mathrm{opt}}:\pi_{1}=1:4$, in which case the advantage of PN-GAIL over other baseline methods is clearly visible. In addition, following your advice, we have added experiments at the ratio of $\pi_{\mathrm{opt}}:\pi_{1}=1:10$ in Common Response point 1. We notice that WGAIL performs even worse than GAIL, which we attribute to the fact that WGAIL incorrectly assigns high confidence scores to non-optimal demonstrations, resulting in a worse policy.

Q2. Modifications of Figures 2, 3, 4.

A2. Thank you for your valuable comments. We have redrawn Figures 2, 3, 4 and also included the results of 2IWIL in Figure 3, following your advice. The redrawn plots have been placed in the revised version.

Q3. How would the results in Figure 2 be affected if $\pi_\mathrm{opt}$ were dominant?

A3. In the Ant-v2 and Pendulum-v1 environments, we supplement the experiments at the demonstration ratio of $\pi_\mathrm{opt}:\pi_1=2:1$ in Common Response point 2.

Q4. What would occur if the optimal demonstrations were kept invariant while the number of suboptimal demonstrations varied?

A4. We have conducted experiments with different numbers of non-optimal demonstrations in Common Response point 3. We find that when the number of non-optimal demonstrations decreases, the performance of PN-GAIL does not decrease significantly.