Stylized Offline Reinforcement Learning: Extracting Diverse High-Quality Behaviors from Heterogeneous Datasets

Q5: All experiments were conducted on environments with discrete action spaces. How does SORL perform in continuous action environments?

A5: Although we don't include environment with continuous action space, our algorithm can be easily adapted to the continuous action space. The adaptation process involves representing action probability generated by the policy network as a continuous distribution (often Gaussian). Our method is not limited to discrete action space. The main contribution of SORL is to propose a novel framework that learns diverse and high-quaity qualities from offline datasets. The distinction between discrete and continuous action spaces in experiments lies outside the primary focus of this paper. We do recognize its importance for future work.

Q6: How was the diversity metric measured for InfoGAIL? To my knowledge, InfoGAIL does not explicitly split the policy into multiple clusters but instead learns a multi-modal policy.

A6: We set the latent space of InfoGAIL as an $m$-dim one-hot vector, where $m$ is the number of styles. For example, when $m=3$, we get the 3 policies using the one-hot latent codes $(0,0,1),(0,1,0),(1,0,0)$.

Q7: The explanation on the "lora" module seems missing.

A7: We appreciate the reviewer's advice to include a explanation of LoRA module [1] used. We have provided a more detailed explanation in Appendix E. We share the main network among policies of all styles and each policy has its own LoRA module.

References:

1.Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

We greatly appreciate your thoughtful review. We have carefully considered the comments and have provided detailed clarification. We appreciate it if you have any further feedback.

Q1: There is no theoretical ground for naively replacing $A_{\mu^{(i)}}$ with $A_\mu$ without importance sampling corrections. At least an empirical ablation study should be provided if it is difficult to devise a theoretical justification.

A1: We conducted a comparison of performance (referred to as quality) using $A_{\mu^{(i)}}$ and $A_\mu$. We employ a soft clustering approach where each trajectory has a probability of belonging to each cluster. During the training of $A_{\mu^{(i)}}$, the dataset trajectories are weighted based on the probability obtained from the soft clustering method. In the experiment:

The empirical results indicate that utilizing $A_{\mu}$ during the learning process enhances the quality of policies. The reason that $A_\mu$ generally outperforms $A_{\mu^{(i)}}$ is that it only focus on part of the dataset.

Q2: It isn't easy to understand the proof presented in Appendix B.

A2: We sincerely appreciate the reviewer for pointing it out. We've refined the proof, which is now in Appendix D.

Q3: The diversity metric proposed by the authors does not consider how different the styles are.

A3: We appreciate the reviewer to point this out. We agree that one limitation of the current metric of diversity is that it relies on the clustering $\hat{p}(z=j|\tau)$. Comparing the diversity of two sets of policies are not trivial. Previous research, such as InfoGAIL, also did not propose a specific metric for evaluating diversity. Instead, they relied on visualization to showcase the diversity of learned policies. In order to provide a more comprehensive representation of the diversity of learned policies, we have proposed additional diversity metrics, under which the described problem does not exist. We have incorporated three additional metrics into our testing: the skill metric, the OT (optimal transport) metric, and the discrimination metric. The skill metric measures the dissimilarity of actions, while the OT metric quantifies the disparity in state distribution. The discrimination metric assesses whether the policies can be distinguished by a neural network. The table below showcases the diversity of different style learning methods in the grid shooting environment, measured using various metrics. Higher values indicate greater diversity. For a detailed description of these metrics, please refer to the Appendix F. Based on the results of the additional diversity metrics presented in Table below, we can conclude that the SORL algorithm is capable of obtaining diverse and high-quality policies. InfoGAIL fails to learn diverse policies, while Off-RLPMM has considerably lower winning rate.

Q4: §5.3 states that the character has to combat against an AI opponent. What is the strategy of the opponent? Is SORL robust to the changes in the opponent's strategy?

A4: The opponent is an agent with a fixed strategy that move randomly, and shoot with a probability once the shooting action cools down.

To check whether the agent is robust to the changes in the opponent's strategy, we introduce a moderately strong opponent with DQN training for 20,000 steps.

Despite the opponent's increased strength, the policies employed by our agent remained stylized. Consequently, the winning rate experienced a decrease compared to previous opponents, but it remained competitive. Specifically, the winning rate of policy 1, which excels at shooting enemies, declined to $19$% due to the increased difficulty of successfully shooting and winning against the stronger opponent. On the other hand, the winning rate of policy 2, which specializes in collecting stars and focuses on escaping from the enemy's shooting, was not significantly affected.

Q4: As the author claims the induced policies are high-performing, the baselines should include some strong offline RL methods for comparison. This will also show SORL's advantage in policy diversity compared to them. The current baselines are too few.

A4: We add experimental results of standard offline RL methods (CQL, AWR and IQL). We conduct comparison in Atari games and results are shown below. Notably, SORL learns a set of policies while the remaining methods learn a single policy. The diversity metric cannot be calculated for methods that only generate a single policy. The scores of SORL are averaged over the full set of learnt policies. We have not yet completed all the experiments on AWR. However, we will finish them and include them in Appendix G before the deadline.

The results above shows that the quality performance of SORL is still generally higher than offline RL baselines. We hypothesize that the human datasets are multi-modal, and utilizing multiple policies instead of a single policy to fit the dataset yields better results.

Q5: How do the authors collect online user data? Where is the user agreement to collect this data? This should appear at least in the appendix.

A5: The game environment and the dataset used in this paper are public. The link to the game environment is https://github.com/FuxiRL/DunkCityDynasty, and the link to the dataset is https://huggingface.co/datasets/FUXI/DunkCityDynasty_Dataset. To our best knowledge, this paper is the first to conduct experiments of learning both diverse and high-quality behaviors from a large human dataset of video games.

We greatly appreciate the valuable feedback provided by the reviewer. We have carefully considered the comments and have addressed them in our responses. If you have any additional questions or concerns, please don't hesitate to let us know.

Q1: Transformation from the true posterior to Eq. 2 needs more explanation.

A1: We appreciate the reviewer for pointing out that 'if the policies only slightly differ from each other, the consecutive multiplication of all steps within the trajectory will make their trajectory distributions very different from each other'. We plot the distribution of $\hat{p}(z=1|\tau)$ for all trajectories in the dataset in Figure 11, Appendix J. Figure 11(a) uses consecutive multiplication, while Figure 11(b) is the average of all steps. The results show that using consecutive multiplication causes a trend of polarization in clustering, which leads to unexpected numerical instability. The experiment was conducted in the grid shooting environment with the number of clusters set to $m=2$. To address this issue, we have chosen to average all the steps instead of multiplication in order to alleviate the problem of numerical instability.

Q2: The proposed SORL needs to know the number of policy primitives constituting the dataset in order to learn the diverse policies. But it is hard to know this prior under many ciucumstances. I think there should be a study about how sensitive SORL is to this hyperparameter.

A2: We appreciate the reviewer for pointing out the importance of conducting sensitivity analysis on the number of policies $m$ in SORL, as it is difficult to know the exact value of $m$ in real circumstances. In order to address this concern, we have conducted experiments to evaluate the diversity, quality, and consistency of SORL under different values of $m$. We also add two more diversity metrics that are unrelated to $m$, and their detailed descriptions can be found in Appendix F. The performance of SORL remains similar across different $m$ values, indicating that the algorithm is robust to the hyperparameter $m$.

Q3: How diverse are the induced policies? The case studies are great but there should be a quantitive study. My further question on this is, if we set m larger than the actual number of policy primitives $\mu_{1,\cdots,m}$ , what will the resulted policies be like?

A3: The quantitative results of the grid shooting environment can be found in the table provided below (also available in Appendix A).

We provide additional results in the grid shooting game as $m$ increases. The actual number of policy primitives is two. The reward distribution of the learned policies is presented in the following table, taking $m=4$ as an example. For brevity, the resulting policies for $m=3$ and $m=5$ can be found in Appendix I.

When taking larger $m$ values, the learned policies still differs a lot from each other. In the case when $m=4$, one policy learns to focus on shooting, one policy concentrates on star collection, and the remaining policies' preferences lie between the above two extremes.

Many thanks for conducting the additional experiments. My concerns have been addressed and I have increased my score.

Q6: Could the authors provide more details about the setting of the Dunk City Dynasty experiments? For example, the action space, and model structure. Appendix C only describes the structure of the first two experiments.

A6: We appreciate the reviewer for pointing it out. We have provided details of the experiments in Dunk City Dynasty in Appendix B. The state space dimension is $468$, including global state, allies' states and enemies' states. The action space is a discrete space with $52$ actions. The network structure of 'Dunk City Dynasty' is a 3-layer MLP. The opponent in evaluation is an agent with moderate strength, that is learned by vanilla Behavioral Cloning for similar training steps (100,000 steps).

Q7: What does it mean by “we share the main network and use a lora module to discriminate different styles.” In Appendix C?

A7: The detailed description of LoRA module is as follows. In order to ensure balanced learning among all the styles, we share the main network and use a LoRA module to discriminate different styles. LoRA [1] is a widely used network structure, that substitutes the original matrix of the linear layer by a matrix and a low-rank multiplication of two other matrices. Further details of LoRA module is added in Appendix E.

Q8: Results in Table 3 show that SORL has a very high variance (5.3 ± 3.4) in terms of quality, which only slightly outperforms InfoGAIL (5.0 ± 0.8). Does this mean pursuing high-quality sacrifices for the performance of the policy? Then, what do we gain from the high diversity?

A8: The focus of our work is to learn diverse and high-quality policies. These are two distinct objectives and pursuing diversity may sacrifice the quality of the policy. The goal of this paper is to maximize both objectives and trade off between diversity and quality. In Table 3, although SORL shows higher variance in performance, it signficantly outperforms the baselines in terms of disversity. Similarly, in the other two environments, we also observe a trade-off between quality and diversity. For instance, Figure 2 in the main text shows that the SORL algorithm does not always outperform the baselines in terms of quality, but it consistently achieves a good balance between high quality and diversity. By combining the pursuit of both quality and diversity, we demonstrate that the SORL algorithm is superior to the baselines. It provides a more comprehensive solution that achieves both high performance and a diverse set of policies.

The motivation of learning diverse policies from offline datasets has been expained in the response to question 2. To recapitulate briefly, it is useful in applications including learning human-like behaviors in game AI and opponent modeling.

References

1.Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.

Q4: The metric for evaluating diversity seems to rely on the learned clustering p.

A4: We admit that the diversity metric employed in the paper relies on clustering $p$. Comparing the diversity of two sets of policies are not trivial. Previous research, such as InfoGAIL, also did not propose a quantitative metric for evaluating diversity when learning policies from datasets. Instead, they relied on visualization to showcase the diversity of learned policies.

In order to provide a more comprehensive investigation of the diversity of learned policies, we have proposed additional diversity metrics that do not require clustering: the skill metric measuring the dissimilarity of the skill set and the OT (optimal transport) metric quantifing the disparity in state distribution. For a detailed description of these metrics, please refer to the Appendix F. Moreover, we will present visualization results of the Atari environment in the response to the next question.

The table provided below showcases the diversity of various style learning methods in the grid shooting environment utilizing these two additional metrics as well as popularity metric used in the original paper. The two new metrics are not dependent on clustering. Higher values indicate greater diversity. Based on the results of the additional diversity metrics presented in Table below, we can conclude that the SORL algorithm is capable of obtaining diverse policies. The results demonstrate that SORL learns a set of diverse and high-quality policies. InfoGAIL fails to learn diverse policies, while Off-RLPMM learns low-quality diverse policies and has considerably lower winning rate.

Q5: Could the authors provide some visualization or example of the mean of the clusters in Atari games? Could the authors plot the shooting positions of each policy? Also, besides the shooting position, are there other differences between these policies?

A5: We have generated visualizations of the clusters in several Atari environments. Detailed descriptions of the environment and the visualization can be found in Appendix H. In SpaceInvaders, the shooter can move horizontally, and there are three stationary bunkers positioned above the shooter. To evaluate the diversity of learned policies, we calculate the frequency of the shooter's position over 30,000+ steps for each style. The table below demonstrates that different styles of policies tend to prefer different positions.

In MsPacman, the map and starting position are fixed. To evaluate the diversity of learned policies, we calculate the frequency of the beginning trajectory until the first corner for each style, based on 30 trajectories per style. There are a total of six possible corners. The table below displays the different preferred corners for each style:

As for the Dunk City Dynasty, the plot of shooting positions are added to Appendix B. Besides the style difference of shooting positions, there also exists the style difference of ball passing and ball possesion, as shown in the following table. Style 2 has the fewest number of passes and the lowest proportion of goals scored directly after a pass.

Thanks for the valuable feedback. We appreciate the reviewer's constructive comments and would like to address your concerns. Please let us know if you have any further questions or comments.

Q1: No standard Offline RL baseline compared. As quality and diversity are both metrics for evaluation, it would be good to compare the performance with other standard offline RL methods, e.g., CQL, TD3+BC, AWR.

A1: We add experimental results of standard offline RL methods (CQL, AWR and IQL). We exclude TD3+BC, because it works for the continuous action setting, different from the experiment settings with discrete action spaces. Instead, we include IQL (Offline Reinforcement Learning with Implicit Q-Learning), which is a strong offline RL algorithm. We conduct comparison in Atari games and results are shown below. Notably, SORL learns a set of policies while the remaining methods learn a single policy. The scores of SORL are averaged over the full set of learnt policies. We have not yet completed all the experiments on AWR. However, we will finish them and include them in Appendix G before the deadline.

The results above show that the quality performance of SORL is still generally higher than offline RL baselines. We hypothesize that this is because the human datasets are multi-modal, and utilizing multiple policies instead of a single policy to fit the dataset yields better results.

Q2: The motivation for increasing the diversity of policy is not clear.

A2: We have discussed the motivation of learning diverse policies from offline datasets in Lines 4-7 in the introduction. Additionally, we have added further content discussing the utilization of learning diverse policies from offline datasets in opponent modeling in Lines 8-10. We appreciate the reviewer for pointing out that the motivation needs a more detailed discussion. Learning diverse policies from offline datasets is beneficial for various real-world applications, such as game AI and opponent modeling. In game AI, offline data collected from human gameplay allows AI bots to mimic human-like behavior. In online games, deploying such AI bots with varyied motion styles can enrich the gaming environment and enhance player engagement. Additionally, in opponent modeling, high-quality diverse opponents that resemble real opponents can significantly improve the performance of the learned policy. It is worth noting that learning diverse policies from offline datasets is an area that has not been extensively explored in previous research. In the related work, we mention encouraging exploration and skill discovery because they are the motivation of promoting diversity in online settings rather than offline settings.

Q3: Many details are missing in the experiment of the “Dunk City Dynasty”. The code does not include this experiment.

A3: We have updated the supplementary materials for "Dunk City Dynasty" and included the algorithm codes. However, please note that the code for large-scale efficient experiments cannot be made public at this time due to institution regulations. We will make it available to the public as much as possible in the future.

We thank the reviewer for the constructive comments, and we provide clarification on your concerns as follows. We would appreciate it if you have any further feedback.

Q1: Will the video game environment and the dataset be released?

A1: The game environment and the dataset used in this paper are public. The link to the game environment is https://github.com/FuxiRL/DunkCityDynasty, and the link to the dataset is https://huggingface.co/datasets/FUXI/DunkCityDynasty_Dataset. To our best knowledge, this paper is the first to conduct experiments of learning both diverse and high-quality behaviors from a large human dataset of video games.

Q2: The connection between the 'style' of play and the 'reward'.

A2: We agree with the reviewer that 'the connection between the 'style' of play and the 'reward' outcome merits a deeper examination'. In our perspective, there are two types of policy diversity: different policies of comparable performances and policies of novice/veteran distinction. In the first case, although the policies are diverse, they may still have similar expected returns. In this paper, we primarily focus on this case. For the latter case, one simple approach is to split the dataset into subsets with respect to the cumulative return of trajectories, and learn different policies on corresponding subsets. In the experiments (Section 5.3), we find that our method discovers two distinct strategies for winning the grid shooting game: shooting the enemy and collecting stars. The motivation of this paper is to enhance the quality concerning the first kind of diversity.

Q3: The simplicity of the clustering model raises concerns regarding its ability to discern more nuanced styles. A more critical discussion on the model's capacity to identify and differentiate between complex styles would enhance the paper.

A3: Regarding to your concerns on its ability to discern more nuanced styles, we acknowledge that the MLP utilized in our current implementation cannot model policies that need memory to characterize styles, since it doesn't memorize historical states. We believe that future work could incoperate network structures with memory, such as transformers and recurrent neural networks (RNNs).

To discuss more on the model's capacity to identify and differentiate between complex styles, we provide an additional experiment on different numbers of styles $m$, especially when $m$ is set larger than the actual number of styles of the dataset.

We have conducted experiments to evaluate the diversity, quality, and consistency of SORL under different values of $m$. We also add two more diversity metrics that are unrelated to $m$, and their detailed descriptions can be found in Appendix F. The results show that the performance of SORL remains consistent across different values of $m$, indicating that the algorithm has sufficient capacity to generate diverse policies.

Q4: Regarding the implementation of AWR, it seems to be applied to each cluster individually.

A4: Our method is scalable with respect to the number of clusters. Firstly, in our algorithm, we employ a soft clustering approach where each trajectory has a probability of belonging to each cluster. This partially mitigates the problem by allowing trajectories to contribute to multiple clusters. Secondly, policies of different styles share a common part of the network parameters, which helps in learning better stylized policies. Further details about the shared network structure can be found in the Appendix E.

Q5: The paper omits a benchmark against "robust imitation of diverse behaviors".

Q6: The paper could explore the potential of employing more advanced offline reinforcement learning algorithms in the second step of the methodology.

A5,6: Thank you for bringing to our attention the benchmark "robust imitation of diverse behaviors" and the potential use of advanced offline RL algorithms in the second step of our methodology. We will definitely consider these suggestions for future improvements.