Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning

We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.

W1: I would like to know if there are other reweighting methods that can be used for comparison. If so, could some comparative experiments be included?

R1: In fact, there are some weighting methods can be compared, inluding: DWBC[1] weights the IL loss using the discriminator, and [2] weights via the old policy). Due to time limited, we only implement DWBC as our competitor. For the DWBC method, the original DWBC algorithm uses $d(s, a, log \pi)$ to determine whether the current sample is generated by the expert policy. In our setting, we use $d(s, a, z)$ to determine whether the current sample belongs to style z. The experiment results is shown in Table 9. We can see that DWBC performes better than BC but worse than BC-PMI. We have add the results in Appendix E.

[1] Xu, Haoran, et al. "Discriminator-weighted offline imitation learning from suboptimal demonstrations." International Conference on Machine Learning. PMLR, 2022.

[2] Sasaki, Fumihiro, and Ryota Yamashina. "Behavioral cloning from noisy demonstrations." International Conference on Learning Representations. 2020.

W2: Additionally, the current method evaluates the diversity of state-action pairs. As far as I understand, there are also studies specifically focused on state diversity and action diversity. I wonder if these could serve as a standard for comparison?

R2: We have incorporated some state diversity (e.g., area style in Atari) and action diversity (e.g., fire rate style in Atari) in our experiment. We believe this reflects the diversity evaluated under state-only or action-only perspectives to some extent. Introducing a more theoretical definition of state or action diversity into the evaluation is indeed an interesting idea [1]. We have further discussed other possible metrics for state diversity or action diversity in the revised version.

[1] Suneel Belkhale, Yuchen Cui, Dorsa Sadigh. Data quality in imitation learning. NeurIPS 2023

Dear Reviewer BUhV,

Thanks again for your valuable review. We are wondering whether our rebuttal has addressed your concerns. We would love an active discussion with you and hope to address any remaining concern or question you may have.

Best Regards, Authors

We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.

W1.1: A key disadvantage of the method is that it requires style markers for each trajectory in the training dataset. These can be expensive to obtain. A brief note is made about using labeling functions to obtain these markers, but no experiments are included.

R1.1: In our experiments, we utilized a labeling function, which takes a minimal amount of time to label the trajectory since it only runs once for each data point. Following common practice, we approached this by specifying a task-relevant style embedding (represented by 𝑧) in advance. In Atari, similar to Wu et al. (2023), we pre-defined styles such as fire rate. In basketball, we defined "destination" and "curvature" as the style labels, consistent with the styles defined in Zhan et al. (2020). We have made this clearer in Appendix D and provided a Python-style labeling function for fire rate to better illustrate this process.

W1.2: This also makes the comparison with existing works unfair since they are unsupervised methods. To make the comparisons fair, the authors should include the supervision in other methods, e.g., they can train the posterior network in InfoGAIL using the style labels.

R1.2: Actually in the our implement of InfoGAIL and SORL, in order to make the comparison more fair, we concatenate the state and style as input to the network. We made this more clearer in the revision of the version (as highligted in Sec 5.1).

[1] Wu, Shuang, et al. "Quality-similar diversity via population based reinforcement learning." The Eleventh International Conference on Learning Representations. 2023.

[2] Zhan, Eric, et al. "Learning calibratable policies using programmatic style-consistency." International Conference on Machine Learning. PMLR, 2020.

W2: The experimental section is relatively brief, lacking analysis on the method’s performance as the number of styles increases or when labels are noisy. The latter case is quite important given that labels generated by programmable labeling functions (as mentioned in L296) may be imperfect. For a comprehensive understanding of these factors, it would be beneficial to include experiments that measure calibration as the number of styles increases, as well as tests assessing robustness to label noise introduced by such labeling functions.

A2: We have conducted experiments and reported the results in Table 7 and Table 8. In Table 7, we compare the calibration for styles with different noise to analyze the impact of noisy sample ratios. In Table 8, we increased the number of styles for the fire rate in Atari and compared the performance of CBC and BC-PMI. The experiments show that in both cases, BC-PMI performs better than CBC. This is expected since PMI weights the imitation learning loss according to mutual information, which implicitly learns the distribution between styles (with noise) and state-action pairs.

Dear Reviewer fKFr,

Thanks again for your valuable review. We are wondering whether our rebuttal has addressed your concerns. We would love an active discussion with you and hope to address any remaining concern or question you may have.

Best Regards, Authors

We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.

W1: This paper aims to enhance policy diversity by incorporating a weighting method based on pointwise mutual information to the Conditional Behavioral Cloning framework. The key proposal of this paper is merely the introduction of a data weighting algorithm, which, in my view, does not represent adequate technical contribution.

R1: We would like to clarify the contributions of our paper as follows:

1. For the task of recovering diverse policies from diverse expert data, to the best of our knowledge, we are the first to make the observation that often only a part of a trajectory is highly relevant to the style of a policy. We provide a solid solution (based on pointwise mutual information between state-action pair (s,a) and style variable z) to quantify this, based on which we develop a novel and principled diversity recovering imitation learning method, i.e., BC-PMI.

2. Our approach is both elegant and effective, because we pay more attention (in a principled way via Pointwise mutual information weighting) to those state-action samples that are more likely generated by a policy with a targeting style, when learning a policy with the style; Moreover, we theorectically unified the BC and CBC as special cases within our framework.

3. We provide an intuitive motivating example (Circle 2D) and verify the effectiveness of our approach in multiple diversity metrics in the image-based environment Atari and real world basketball datasets.

W2.1: The manuscript does not include sufficient ablation experiments to validate the efficacy of the proposed data weighting method that utilizes pointwise mutual information. It remains unclear how BC-PMI compares to Conditional Behavioral Cloning that is conditioned on style, where each sample point is assigned equal weight.

R2.1: Thank you for highlighting the need for additional ablation experiments to further validate the efficacy of our proposed data weighting method. We would like to point out that we have provided a comparison between BC-PMI and CBC in both the Atari and basketball benchmarks (please refer to Table 3 and Table 4), and we provides more ablation experiments in Table 8 and Table 9 as suggested by reviewers. Additionally, we wish to emphasize that Conditional Behavioral Cloning (CBC) can be viewed as a special case of our BC-PMI method. In CBC, each state-action pair (s, a) is assigned an equal weight of 1 across all style labels, which means it does not differentiate between the relevance of different samples to the style being learned. In contrast, our PMI method adjusts the weights according to the specific relevance of each sample to the style, thereby enhancing the learning process by focusing on the most informative samples. We further explain this in Fig 7 in Appendix C.

W2.2: Additionally, its performance against other traditional data weighting methods has not been adequately compared.

R2.2: Due to time limited, we implement DWBC [1] as our competitor. (There are also other candidates such as [2]). For the DWBC method, the original DWBC algorithm uses $d(s, a, log \pi)$ to determine whether the current sample is generated by the expert policy. In our setting, we use $d(s, a, z)$ to determine whether the current sample belongs to style z. The experiment results is shown in Table 9. We can see that DWBC performes better than BC but worse than BC-PMI (Ours). We have add the results in Appendix E.

[1] Xu, Haoran, et al. "Discriminator-weighted offline imitation learning from suboptimal demonstrations." International Conference on Machine Learning. PMLR, 2022.

[2] Sasaki, Fumihiro, and Ryota Yamashina. "Behavioral cloning from noisy demonstrations." International Conference on Learning Representations. 2020.

Dear Reviewer 6kbz,

Thanks again for your valuable review. We are wondering whether our rebuttal has addressed your concerns. We would love an active discussion with you and hope to address any remaining concern or question you may have.

Best Regards, Authors

Thank you for the detailed explanations and significant effort in addressing my concerns, which have resolved most of my points. I encourage the authors to validate the effectiveness and applicability of PMI on a wider range of advanced BC policies when time permits in the future. Considering the overall improvements, I have decided to increase my rating.

We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.

W1: Given that stylized experts make unbalanced trajectory generation in training, it might be helpful to discuss the disentanglement between the stylized code and the state-action pair. Some previous works have built related benchmark for the purpose such as [1] and [2]. Though the benchmarks may not be originally designed for imitation learning, the corresponding metrics might be worth reference. Such evaluation can be adapted by measuring the mutual information or other metrics between the stylized code and the pair.

R1: After reviewing the methods in [1, 2], we acknowledge that we can adapt $R_{ij}$ and MED to weight the BC loss. However, we still need PMI instead of MI for disentanglement, as our focus is on state-action pairs. Therefore, it is possible to adapt the methods you referred to by replacing MI with PMI in the calculation of $R_{ij}$ and MED. We have included a discussion on the possible improvements through combining our method with other disentanglement methods in the revised version.

[1] “An Empirical Study on Disentanglement of Negative-free Contrastive Learning”, NeurIPS 2022.

[2] “Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations.”, ICML 2019

W2: It remains not unclear, at least not generally convincing across datasets, that PMI shows a significant advance than the usual MI for BC. More evidence or discussion about this can be helpful to enhance the significance of the proposed method as PMI is claimed as a main contribution in this paper. If replacing MI with PMI can result in generalizable performance boosting across datasets, the proposed method can be supported with more experimental significance.

R2: For our experiments, we utilized the Atari and basketball datasets. The Atari dataset represents a complex environment with discrete action spaces in a virtual gaming scenario. In contrast, the basketball dataset is a large, real-world dataset consisting of 500k data points and features a continuous action space. Therefore, we believe that the benchmarks we have chosen demonstrate the generalizability of our method. In future work, we will consider involving other benchmarks (e.g. MuJoCo) to further validate our approach.

Dear Reviewer ihL7,

Thanks again for your valuable review. We are wondering whether our rebuttal has addressed your concerns. We would love an active discussion with you and hope to address any remaining concern or question you may have.

Best Regards, Authors