Generative Modeling of Individual Behavior at Scale

Thank you for your review. Regarding our contributions being limited to stylistic adaptation in gaming, Chess and Rocket League are very different games, despite being within the gaming domain. Additionally, the way we train Rocket League is not so different from how language models are trained (we use GPT-2 [2] as our base model). As a result, we see our methodology generalizing fairly easily to any domain where there is a partitioning of the dataset (e.g., into individual players/people). As a more general application of our methodology to a domain other than games, we have applied our method to an image generation setting using diffusion models, as described in our general response above.

Regarding the comparisons to McIlroy-Young et al, we acquired the splits directly from the authors of the original papers for the 400 players stylometry test. However, we were unable to access the sampling method used to create the larger dataset for their unseen tests. As a compromise, we used the same exact player filtering criteria and the same timeframe to source the larger 10,000 player dataset that we used in our few-shot test. This evaluation is significantly more difficult, as it contains thousands of additional players that are trained independently of each other (recall that the LoRA matrices are frozen during few-shot training) and should be a superset of the original dataset. For this reason, we believe that this comparison is more than fair, as we test on a substantially more difficult task and improve significantly on their results.

We address your questions below.

1. Please see above.

2. We can use steering to change a model’s behavior after training it. This is especially useful when there are behaviors that a user may want to change. For example, in chess, if a player is struggling against players who use their bishops effectively, a model that uses bishops more frequently can be generated without training by adding the corresponding style vector. More generally, if a user desires a different distribution of outputs, our method can be used to drive the model closer to that desired distribution, similarly to what fine-tuning would do. The major difference between steering and fine-tuning, however, is that there are no gradient calculations required after MHR has been trained. This makes the process instantaneous and cheap.

3. We discuss this in the appendix A.4. Additionally, the players in Figure 3 will have sets of data that are not necessarily from the same timeframe, yet still show high similarity to other vectors from the same player. We additionally train very few-shot vectors in Figure 9 of the appendix, which show that the similarity to a vector trained on all games is still several standard deviations above the mean. A split of 25 games may not contain all variations of a player’s playing style, but still contains enough information to learn the general tendencies of a player, even across time. Indeed, the P-value for the similarity of a vector trained on even just 10 games (roughly 2 standard deviations above population mean) is about 0.0455, which is statistically significant.

[1] Y. Zhang, A. P. Jacob, V. Lai, D. Fried, and D. Ippolito, Human-aligned Chess with a Bit of Search, Arxiv 2024.

[2] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language Models are Unsupervised Multitask Learners, 2019.

Thank you for the additional experiments, particularly the application to image generation. This demonstrates potential generalizability! However, I am still concerned that the effectiveness of the method hasn't been completely demonstrated. In particular:

• While you highlight extensions to image generation, the paper does not compare your method against established baselines such as CLIP-guided editing, DreamBooth, or StarGAN. Demonstrating superiority or complementarity to such methods would significantly strengthen the claims of broader applicability.
• Regarding the McIlroy-Young et al. evaluation, your defense about dataset construction is reasonable. However, to ensure fairness and comparability, applying their model to your new test set is necessary. Reporting their original scores on a different test set does not provide a direct comparison and risks misinterpretation.

These details about the test set with respect to McIlroy-Young et al. seem rather important, and I'm wondering why these details were not included in the original paper?

Thank you for your response! We address the generalizability point and the comparison to McIlroy-Young et al. below.

The purpose of implementing the image generation example was to show the generalizability of our adapter framework and style steering method to a new domain. Although we do not have enough time during the ICLR discussion period to evaluate all of the alternative methods you mentioned, we believe our approach can complement these methods while enabling finer control. Currently, we are able to compare against DreamBooth using LoRA by leveraging existing online scripts [1]. We find that our method allows for more granular control in the final image than DreamBooth, while also retaining more of the original image. You can view this comparison in the link below; we have also added it to the supplementary PDF and have updated our general response above.

Link to DreamBooth comparison samples: https://imgur.com/a/9teIqAk.

Regarding applying our method to domains outside of gaming, we will add discussion of concrete examples to the paper. For example, our methodology can be used in the following applications: in image generation, performing fine-grained editing of an image (as we have demonstrated); in LLMs, mitigating or emphasizing stylistic aspects of the generated text; in racing, predicting how human drivers will respond to changes in the car’s aerodynamics (this is part of our ongoing work).

Regarding the comparison to McIlroy-Young et al. for unseen few-shot players, we note that both their base model and ours were trained on the same exact dataset of 400 players. For testing, we reached out to the authors asking for both the model and the test set, but the authors were unable to provide them. As a result, we performed a test that we believe is strictly more difficult, over a much larger universe of 10K players. Note that McIlroy-Young et al.’s method can only perform stylometry — it cannot create generative models of individual behavior or allow style steering, which is the main goal of our work. In order to directly compare the stylometry numbers, one thing we could do is to recreate McIlroy-Young et al.’s model and evaluate it on our test set of players. We cannot complete this in time for the ICLR discussion period, but we can certainly do so before the camera-ready deadline (and will also reach out again to the authors). We will also update Table 1 and our caption/text description to accurately reflect the comparison we did, so that there is no room for misinterpretation.

If you have any further questions, please feel free to ask.

[1] https://github.com/huggingface/peft/blob/main/examples/lora_dreambooth/train_dreambooth.py

Thank you for your review. Indeed, the potential of applying MHR fine-tuning and style steering to other domains is an exciting prospect!

As you observed, we do not claim novelty over the PEFT techniques (Polytropon, MHR) used in this paper, which are typically used to train low-rank adapters for multi-task learning in NLP. Instead, we believe our novelty comes from our application and adaptation of a multi-task adapter framework to behavioral modeling and stylistic analysis. We find that these adapters create an interpretable and steerable style space, which we validate through chess and Rocket League, as well as the image generation example shown in our general response. Our style methodology and style delta vector method in particular enable new capabilities, such as the ability to create new styles and steer (change) existing styles in a human-interpretable way.

Regarding a user study, we agree that this would be a very interesting and relevant investigation. In our paper, we have reported a quantitative set of heuristics (that were designed by chess masters and software engineers) to characterize playing style, though it would be interesting to see how humans perceive these changes. We have consulted with chess masters in the past to validate the effect of steering a player’s style vector, comparing moves made by the player’s model before and after the change. However, we have found that the quantitative heuristics allow us to validate the style change more effectively at scale.

We do not provide such a steering analysis for Rocket League due to computational constraints in the environment rollouts, and a lack of single-step heuristics from data points, as many of the existing heuristics depend on behaviors over longer trajectories. It is difficult to generate a reliable and consistent benchmark for this game as it would require hundreds of thousands of rollouts per player in a simulator that can only handle on the order of hundreds of frames per second. As a more general application of our methodology to a domain other than games, we have applied our method to an image generation setting using diffusion models, as described in our general response above.

Regarding few-shot performance, please refer to Figure 2, where we show move-matching performance relative to the number of games available when training a model.

Regarding your questions,

1. The bold numbers show which of the methods perform better in that sub-task. For all of the shown tasks, we outperform previous work.
2. The y-axis in figure 7 is the difference (in standard deviation) between the model’s behavior before and after modifying the model parameters. For example, for the “bishop pair” numbers, we first note the behavior of the model before the modification. We then standardize this number over all players ((feature - feature_mean) / feature_std). We then apply the bishop delta vector by adding to the parameters of the model. We then roll the model out on the same set of positions and see what changed. After, we standardize with the original model’s std and mean ((feature_changed - feature_mean) / feature_std) and subtract from the original numbers. This delta is the change in the model behavior after a change in the model parameters.

We hope the responses above address your concerns. Please let us know if you have any further questions.

Thank you for your detailed feedback. We have integrated your comments into the submission. Regarding lagging performance, in Figure 2, we observe that our method performs very similarly to state-of-the-art behavior cloning methods, while being around 2 orders of magnitude more efficient (see lines 365-369 of our paper). The difference in move-matching accuracy is very minimal across all game counts, and within the margin of error. In other words, we are much more efficient for similar performance, which is crucial for scaling.

We provide answers to your questions below.

1. For Table 1, there are 100 games for few-shot learning for the larger player sets. We test with as few as 25 games in other charts, such as in Figure 2 and Figure 9.

2. For the 400 players stylometry test, we acquired the splits directly from the authors of the original papers. However, we were unable to access the sampling method used to create the larger dataset for their unseen tests. As a compromise, we used the same exact player filtering criteria and the same timeframe to source the larger 10,000 player dataset that we used in our few-shot test. This evaluation is significantly more difficult, as it contains thousands of additional players that are trained independently of each other (recall that the LoRA matrices are frozen during few-shot training) and should be a superset of the original dataset. For this reason, we believe that this comparison is more than fair, as we test on a substantially more difficult task and improve significantly on their results.

3. Fully fine-tuned models of individual behavior (McIlroy-Young 2022b) are too computationally expensive to train at scale (please see lines 364-367 in Section 5.1 of our updated submission). That paper does not report any results in the unseen few-shot setting on a larger player set. McIlroy-Young et al. (2021) creates a player embedding space, which is more scalable in the stylometry setting, but does not yield individual models of player behavior.

4. Random (%) represents the naive baseline of selecting a player at random for the stylometry task. It is supposed to be a measure of how difficult that stylometry task is relative to the existing set of players.

5. The performance of MHR-Maia decreases relative to the individual models by a very small margin, especially considering that there are relatively few players with over 10,000 games across the Lichess platform. One possible explanation behind this difference is that MHR-Maia only has ~256 parameters to store all the skills for an individual player when fine-tuning towards them. This is in comparison to the full model which has several million. We will move the performance discussion earlier into the paper where Figure 2 is introduced. We predict that increasing the rank and skill count of the MHR adapters will bring us in parity or exceed individual fine-tuning, but that may reduce the interpretability of this space. This line of analysis (dimensionality vs. interpretability) would be an interesting line of future research.

Please let us know if you have any further questions or comments.

Dear reviewer 67Ho,

We hope you have had a chance to review our responses to your individual concerns, as well as our general response above regarding the generalizability of our approach. We have added a new experiment in a new domain (image generation) that should address many of your concerns.

If you have any further questions we can answer to help you revise your assessment of our paper, please let us know. Your consideration of our responses and the discussion we've had with other reviewers is much appreciated.

Thank you!

Thank you for your review. We address each question below.

1. As mentioned in our introduction, modeling human behavior in games has many potential applications, including identifying weaknesses in a human’s play and developing personalized training material, developing better AI partners or teammates [5], and creating more realistic or enjoyable playing experiences [6]. The Maia chess bots on Lichess [6] are extremely popular: they have played millions of games with humans (orders of magnitude more than any other bots). However, human behavioral modeling has also been used in other domains, such as in economics to automate supply chain decision making [1], Formula 1 racing to predict how human drivers will respond to changes in the car’s aerodynamics*, and legal cases to provide more reliable models of judges’ decisions—showing that it has broader uses in other fields.

2. The proposed methods in our paper show that a black-box adapter framework inspired by multi-task learning in NLP can be applied to a domain like gaming, allowing us to learn individualized models of player behavior. In addition, our method learns consistent features about each player that can be interpreted and steered in a black-box manner. As far as we know, this is the first type of work that is capable of doing this. We also show a more general application of our method to steer the output of a diffusion model in a fine-grained manner (see general response above).

3. The base models along with their exact dimensionalities and parameter counts are detailed in Section 3.1 of the paper. For chess, we use a Squeeze and Excitation Residual Network (SEResNet) [2]. For Rocket League, we use GPT-2 [3] initialized and trained from scratch with a linear output map to the action space at each encoded state.

4. We unfortunately do not have a baseline with GPT-4o. Prompting an LLM with historical games from an individual player to generate each move in a game would be prohibitively expensive. Nevertheless, a recent paper [4] provides a similar (non-individualized) baseline using GPT 3.5, and shows 53.7% accuracy. They do not fine-tune the model to individual players, though the accuracy they report is similar to other base models before fine-tuning. Their work also suggests that this domain is very data limited, rather than parameter limited.

We hope this was helpful. Please let us know if you have any further questions.

[1] D. Kurian, V. Pillai, J. Gautham, and A. Raut, Data-driven imitation learning-based approach for order size determination in supply chains. EJIE 2023.

[2] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, Squeeze-and-Excitation Networks. CVPR 2019.

[3] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language Models are Unsupervised Multitask Learners, 2019.

[4] Y. Zhang, A. P. Jacob, V. Lai, D. Fried, and D. Ippolito, Human-aligned Chess with a Bit of Search, Arxiv 2024.

[5] K. Hamade, R. McIlroy-Young, S. Sen, J. Kleinberg, and A. Anderson, Designing Skill-Compatible AI: Methodologies and Frameworks in Chess. Arxiv, 2024.

[6] Lichess bots page. Lichess, 2024. Available: https://lichess.org/player/bots

*Ongoing work by authors, blinded for anonymity

I would also like to push back against point 2, that only showing the method works for two game domains is a weakness of the paper.

Game-playing agents are perhaps the most classical application of AI, with research dating to the 1960s on building AI for chess. This is the first paper I've seen to develop a chesbot that tries to efficiently model and replicate individual user behaviour. Even if the proposed method didn't have other applicability in other domains (which I don't think is true), this contribution is worth sharing with the community.

Dear reviewer Ma2G,

We hope you have had a chance to review our responses to your individual concerns, as well as our general response above regarding the generalizability of our approach. We have added a new experiment in a new domain (image generation) that should address many of your concerns.

If you have any further questions we can answer to help you revise your assessment of our paper, please let us know. Your consideration of our responses and the discussion we've had with other reviewers is much appreciated.

Thank you!