Questions

• What other baselines could be used for comparison in these games?

  • Within the datasets provided by Playstyle Distance, it is challenging to identify more comparable baselines due to the lack of playstyle labels in the training datasets. Additionally, the dataset for TORCS and the Atari games contains only one playstyle, which makes contrastive learning impractical.

  • The empirical evaluations focus on ablations of the existing metric, but do not compare against any other metrics. This makes it difficult to assess how the current approach fares compared to alternative ways of conceptualizing playstyle measurement.

    • Regarding the empirical evaluations focusing on ablations of the existing metric and not comparing against other metrics, we note that a possible unsupervised clustering method mentioned in Playstyle Distance shows less than 30% accuracy for a 5-class classification (compared to over 80% accuracy for Playstyle Distance). Playstyle Distance itself is claimed as the first metric of its kind for video game playstyles in its abstract, which is why we focused on improving this baseline.

• Table 1:

  • Can multiple seeds or variations of the method be run to quantify uncertainty in the performance? The text claims multiscale features ("mix") are superior. Looking at RGSK this result is not clear: the result for t=1 is quite close to mix for t=1. For TORCS the mix results are more clearly better (modulo the lack of any estimate of uncertainty).

    • Our experiments, including those using HSD, were conducted with three different HSD encoders, as provided in the official release of Playstyle Distance on PapersWithCode. Below is a detailed table with results from these three HSD encoders, including the corresponding standard deviations. If more detailed standard deviation data for each accuracy in the 100-round random subsampling is needed, we can conduct further experiments, though this will require additional time.

• Better results on 1 of 2 cases evaluated (TORCS) is not very strong evidence for superiority of the method. Perhaps this would be stronger with evidence from Atari as well?

  • The result shows that the baseline is sensitive to hyperparameters, and the hierarchical state version appears more stable in different scenarios.

• Figure 3:

  • As with table 1, is it possible to add uncertainty estimates to the results presented?

    • The curves in Figure 3 include very narrow shaded regions representing the minimum and maximum values from the three HSD accuracies.

  • The results on probabilistic similarity again show clear results for TORCS, but more ambiguous outcomes for RGSK. Lacking any confidence intervals, the results in RGSK look very close to one another for all methods. Is it possible the dataset is the root cause of these issues? TORCS is from AI agents, which may be easier to classify in style than humans (though it's not obvious a priori why this would be true).

    • Regarding the results on probabilistic similarity, it's true that the improvement on RGSK is minor. In TORCS, each playstyle is closely controlled, with slight variations in driving target speed and noise, necessitating a more precise playstyle metric.

• Figure 4: What are the shaded areas?

  • The shaded areas represent the minimum and maximum values of the evaluation results from the three HSD encoders. These models were trained from different random seeds, as released in the official implementation of the Playstyle Distance paper on PapersWithCode.

Questions

• Is Playstyle Similarty a metric?
  • Yes, Playstyle Similarity is a metric comprising several aspects:
    • Playstyle Intersection Similarity: Measures action distribution similarity in the intersection set.
    • Playstyle Jaccard Index: Measures similarity across all observed states.
    • Playstyle Similarity: Encompasses all observed state-action pairs for a comprehensive metric.

Concerns

• The theoretical contribution is marginal, including the connection to video game play. The concept of "Playstyle Distance" itself does not seem to be specific to game play---it is a straightforward distance between policies using encoded state/action pairs (while "straightforward" is a feature (and not a bug) here, it raises the question as to why this paper needs to be framed in this language at all).

  • Our distance metric still relies on the Wasserstein distance for measuring the effort required to change playstyles. The Bhattacharyya distance, as a variant, aligns with our proposed probability mapping function derived from the Weber-Fechner law (as detailed in Section A.1).

• As "Playstyle Distance" and related concepts are previous work, the additional theoretical contributions are minimal. Incorporating the Bhattacharyya distance is fine, but the connections made to human psychology are a bit flimsy. Additionally, the "Playstyle Similarity" metric is ad-hoc (which is isn't inherently bad); it's also not immediately clear to me (or at least it wasn't argued) if it's a metric, as the product of two metrics isn't a metric in general.

  • Playstyle Similarity is indeed a metric, incorporating a conditional consideration of intersection over union. In Section 5.3, we explain that without the Jaccard index, the metric focuses only on intersection states, ignoring samples outside this intersection set. With only the Jaccard index, action distribution is not measured. Thus, it's not merely a product of two metrics, but a combination that leverages all observed state-action pairs, including:
    • Playstyle Intersection Similarity: A metric of action distribution similarity in the intersection set.
    • Playstyle Jaccard Index: A metric of all observed states.
    • Playstyle Similarity: A comprehensive metric of all observed state-action pairs.

• Given the above points, the experimental section is not substantive enough. While the authors compare Playstyle Distance against their various improvements, they use classification accuracy to compare. However, taking a step back---this is now just a supervised learning problem. Why are any of these needed at all? If the actual task is playstyle classification, then algorithms need to be compared against supervised learning approaches.

  • We use a classification task with a similarity metric to validate these metrics. In these cases, it can be assumed that all supervised learning approaches would exhibit random-level accuracy since there are no playstyle labels in the training dataset. Additionally, contrastive learning is impractical for the TORCS and Atari games datasets as they contain only one playstyle, making it unsuitable for such learning.

• While I appreciate the fact that code was provided, there weren't good instructions in the supplementary material zip file. Clicking the PapersWithCode link in the paper then brought me to a page that clearly has the author names and affiliations listed.

  • If you encounter any issues with running our code in the Supplementary Material, we are available to assist. The PapersWithCode link in Section 4 directs to the official dataset and codes of "Playstyle Distance," which serves as our baseline. Our implementation is a clone from their Git repository, and we have ensured not to reveal our names and affiliations.

Concerns

• It is worth noting that Lin et al. (2021) were able to distinguish intersection states even with unprocessed screen pixels in Atari games. - Seems to state "In two different datasets, Lin et al. (2021) did find pixel-identical screens"?

  • In Atari games, pixel-identical consecutive screens can exist in two game trajectories due to the simplicity of the game environment and low randomness. In common DRL applications, we often find almost no repeated visited states due to high-dimensional observations.

• In a different scenario, when treating each state as equivalent, we can invariably pinpoint an intersection state. - Relates to identical, not just equivalent?

  • When discretized states are considered, they can be treated as identical. However, when a state has a corresponding action, we do not eliminate the same state-action pair for constructing the sampling action distribution.

• Unclear what exactly is new in Sec. 3.1 and what was introduced as part of HDS (Lin et al. (2021)).

  • Lin et al. (2021) used a single discrete state mapping and a sample count threshold to discard states with unstable action distributions. They did not use the hierarchical design in HSD for Playstyle Distance as it prefers stable action distributions and not very large discrete state spaces. Our new method in Section 3.1 enables the simultaneous use of different discrete state mappings.

• While distance is a common metric for determining similarity, a larger distance value conveys primarily that two entities are different, without giving much insight into the degree of their similarity - Why doesn't a larger distance relates to a smaller degree of similarity? Drawing from the concept of similarity, we can infer that a smaller distance provides more definitive information about the similarity - Needs to be explained.

  • As the distance increases, the number of potential candidates with the same distance also increases significantly. In a 2D space, these candidates form a circle around a reference point; as the distance increases, the perimeter of the circle increases. Therefore, we focus more on the similar parts for a more definitive prediction.

• General wording: "intersecting states/samples" suggest a partial equivalence of a single state/sample. E.g. "intersecting set of states/samples" could be more appropriate. This is especially relevant, because the HSD model can be used to defined a intersection over (single) states (based on the hierarchy), and therefore the concepts are not clearly distinguishable.

  • We can edit this part for a more precise description as long as the page limit is not exceeded in the camera reday version.

• Sec 4.2.: "Space size" seems not to be introduced - probably number of discrete states?

  • The term "Space size" refers to the maximum count of different states in the discrete representation. We will use the term "state space" for more precise descriptions.

• To evaluate the efficacy of the proposed multiscale state space and to compare it fairly with Playstyle Distance, we primarily focus on the TORCS and RGSK platforms - Needs to be explained.

  • We validate Playstyle Distance with solely multiscale state space on TORCS and RGSK to make sure multiscale state space is effective. For Atari games, single state space version already has a high accuracy with 512 sample size (over 90% accuracy). With the page limit, we move this part with curves to appendix (Figure 11 and 12).

• In this section, we perform a comprehensive evaluation of various metrics, including leveraging full data with union operations. - Seems to state, that state samples from all games are used as a single set?

  • In Sections 5.1 and 5.2, our focus was on the observed state in the intersecting set of two datasets. In Section 5.3, we expanded this to include all observed states. The introduction of the Jaccard index allows us to define action distribution similarity even outside the intersecting set of samples.

• This is based on the assumption that variations in game content can be interpreted as different states - This statement was not empirically evaluated?

  • This assumption was validated using the Atari Console, employing a discrete encoder from other games.

• The "game-merging" study in Sec 5.3. is an interesting piece of additional information, but the per-game results are potentially more relevant. The results should be added to the main paper and in case of space constraints, one may think about rank metrics or a table with just the most interesting dataset sizes.

  • The per-game results of Atari games in Section 5.3 have been moved to Section A.3. We can add a table in the appendix to report accuracy values. If necessary, we can consider removing or moving parts of the main paper to accommodate this table.

• Why was Playstyle Similarity not evaluated in Sec. 5.2?

  • In Section 5.2, our focus remained on states within the intersection. The concept of Playstyle Similarity, which includes the Jaccard index, is discussed in Section 5.3, as it encompasses all observed samples.

• Most importantly, additional comparison baselines from related work should be added. E.g. clustering or supervised methods should be applicable.

  • Common methods like the Fréchet Inception Distance (FID) use latent feature distribution distances, which have been deemed impractical for classification in TORCS (as shown in Playstyle Distance). Supervised or contrastive learning baselines are challenging due to the absence of playstyle labels in training datasets or the presence of only one playstyle in the dataset.

• The evaluation is only performed via classification, but a similarity metric should preserve a distance relation beyond "Top-1". Therefore, a ranking/continuous study should be performed. E.g. by creating or ordering existing playstyles along a continuous spectrum (like passive/aggressive driving) and evaluating the correlation.

  • We plan to use the TORCS dataset for an experiment evaluating continuous spectrum playstyles (5 target speeds from 60 to 80, and 5 levels of noise). This experiment will be conducted during the rebuttal period.

• Sec 5.2 a, b has nearly indistinguishable intervals. A tabular presentation or non-linear graph scaling should be used to improve clarity.

  • We will include tables in the appendix to provide a clearer presentation of the results from Figure 3(a)(b).

Questions

• What are the real use-cases that the identification of playstyle is targeting?

  • Our metrics are designed to concretely define the similarity and diversity of decision-making styles (playstyles). They are useful in validating the diversity of AI policies, assessing whether imitation learning fits a given playing dataset, and for human behavior modeling. For instance, they can detect the similarity of a Go player's playstyle to that of the superhuman AlphaZero AI to prevent cheating. Playstyle metrics are also useful in recommendation systems for identifying potential customers with similar or new playstyles.

• Why is it an unsupervised problem and not a supervised one ?

  • The metric and discrete representation are unsupervised, meaning no playstyle labels are used in training. This approach is akin to zero-shot classification, with no learning how to classify during the discrete representation learning phase. States are used for identifying comparable states for action distribution comparison, which implies playstyle information.
  • According to Appendix D of the HSD paper (Playstyle Distance), it is impractical to perform supervised learning on training the discrete representation or playstyle classification, as either the datasets are sampled from different sources (TORCS and RGSK) or there is only one playstyle (Atari games).
  • Besides, as described in Section 2.1, playstyle classification can be trained by supervised learning, but it relies on predefined labels and cannot handle playstyles not in the training data (the datasets we used cannot be solved by supervised learning).

• How the playstyle references dataset are built and is it realistic?

  • The datasets used in playstyle analysis consist of gameplay collected by rule-based agents, human players, and DRL agents. For performing few-sample analysis, we use random subsampling without replacement, as shown in Figure 2(d).
  • As for the realism of the games used in our experiments, TORCS and Atari games are common in DRL research. RGSK is a racing game available on the Unity Asset Store (https://assetstore.unity.com/packages/templates/systems/racing-game-starter-kit-22615), designed for human players.

• What is the effect of the state discretization technique that is used?

  • The state discretization technique identifies comparable states for comparing action distributions. Without state discretization, it is challenging to measure the action distribution distance directly from playing datasets (observation-action pairs).
  • In Section A.2, our experiments show that the state discretization technique we used (HSD) performs better than downsampling. The learned discrete representation focuses on features more related to gameplay rather than arbitrary features.

• How do you tune the multiscale approach ? Since you are only using few datasets, the way you tune it may be overfitting the datasets, isn't it?

  • We do not tune the multiscale approach, as it does not require any hyperparameters for our proposed metric. The elements that can be tuned are the discrete state mappings. However, the scope of this paper is not about training new discrete representations; therefore, we directly use the trained models released by the baseline method (Playstyle Distance).

• How do you deal with continuous actions since your work focuses on discrete action spaces?

  • Our metrics can handle both discrete and continuous action spaces, with the difference being the adoption of different types of probability distribution distance metrics. Specifically, TORCS has a continuous action space, while RGSK and Atari games have discrete action spaces, as described in Section 4.1.

Concerns

In summary

• Our distance is still based on Wasserstein distance for measuring the effort between changing playstyle. Bhattacharyya distance is a variant that meets our proposed probability mapping function derived form Weber-Fechner law (Section A.1).

In weakness

• The objective of this paper is to define a similarity metric. The experiments for validating this metric involve identification/classification tasks and the application of diversity measurement in DRL agents. The reference datasets used in the experiments serve as examples of targeted playstyles. For instance, to measure an unknown playstyle, we can compare it against known playstyles and identify the most similar one. This scenario is akin to a zero-shot classification task, where no classification task occurs during the training of discrete representations.
• For new games without existing reference sets, it's possible to first collect some trajectories and label them manually (giving meaning to the playstyle) or use a similarity threshold to create reference datasets.
• Regarding concerns about very few intersection states (as in chess), this highlights why discrete state representation is important for determining comparable cases. If the playstyle (decision-making style) is indeed different, it should show action distribution differences in those intersection states (like opening or initial board). If the playstyle is very similar yet leads to different states, the result is either influenced by environmental randomness or decisions made by other players, which are not under the control of the player we are measuring. Otherwise, the same action in the same state should lead to the same next state, making it unlikely to have very few intersection states.
• Regarding the concern about continuous action spaces, TORCS, as described in Section 4.1, is a game with a continuous action space.
• The multiscale approach is rooted in human cognition and involves several levels of granularity. In Table 1 and Section A.2, we demonstrate that each state mapping function contributes some information to playstyle measurement (accuracy > $\frac{1}{\text{number of playstyles}}$). The combined version further improves accuracy (exceeding the use of just one state mapping). As this research is not focused on training discrete states, we directly utilize the hierarchical states from $*Playstyle Distance*$ and also test with downsampling discretization. Our experiments show that more state mappings lead to more accurate predictions. These mappings can be obtained by training more discrete encoders or leveraging HSD (simultaneous training of several discrete encoders with customized granularity).
• Regarding the concern about generality, our metric is built on general decision-making and tested on rule-based agents, human players, and DRL agents. Our experiments are performed on video games, and we claim applicability to video games due to the straightforward scenario of demonstrating multiple playstyles. Should open benchmarks for decision-making styles become available, we believe our approach would be suitable for claiming broader generality. These decision-making problems could include robot control, natural language processing, and others, as described in the introduction, where DRL is a solution for general decision-making.

Test2 (from center)

Compute similarity between the target playstyle, Speed70N2 (driving with target speed 70 and (0.03,0.015) action noise), and all playstyles (a total of 25 playstyles).

• Playstyle Distance (2^20): consistent count = 2

• Playstyle Distance (mix): consistent count = 2

• Playstyle Intersection Similarity (mix): consistent count = 2

• Playstyle Jaccard Index (mix): consistent count = 1

• Playstyle Similarity (mix): consistent count = 3

The continuous playstyle spectrum experiments have been completed. These results will be added to a new section of the appendix if any reviewer agrees that it can help increase the quality of this paper. More detailed experiments about the continuous playstyle spectrum may be conducted for the final version (camera-ready version).

Experiment Settings

Goal: To analyze whether the metric values (similarity) consistently change as the continuous playstyle changes.

Game: TORCS Discrete Representation: The first models of HSD. (There are a total of 3 models.) Dataset:

• Conducted 100 rounds of random subsampling, with each sampled dataset being disjoint and consisting of 512 observation-action pairs.
• According to the baseline paper (Playstyle Distance), there are 5 levels of target speeds (60, 65, 70, 75, and 80) and 5 levels of action noises applied to the 2-dimensional actions: ((0.01,0.005), (0.02,0.01), (0.03,0.015), (0.04,0.02), and (0.05,0.025)).

Compared Metrics:

• Playstyle Distance (discrete state space: 2^20, the baseline method)
• Playstyle Distance (discrete state space: mix)
• Playstyle Intersection Similarity (discrete state space: mix)
• Playstyle Jaccard Index (discrete state space: mix)
• Playstyle Similarity (discrete state space: mix)

If the similarity consistently decreases as the continuous playstyle changes significantly, we mark a (C) for counting the consistency of a row or column of the playstyle. The higher the consistency count, the better. We also record the standard deviation of metric values in 100 rounds of random subsampling, denoted as (std) in the table.

If the similarity is strictly decreased as the continuous playstyle changed more, we mark a (C) for counting the consistency of a row or column of the playstyle. The higher the consistency count, the better. We also record the standard deviation of metric values in 100 rounds of random subsampling, denoted as (std) in the table.

Hello Reviewers,

The new experiments involving additional gaming datasets have been completed. These datasets are unstable and include noise, therefore, we also compared the expected version of playstyle metrics (measures) in addition to the uniform version presented in our main paper. If applicable, we will add a new appendix section (A.5) in the camera-ready version to describe these variants. The primary difference lies in how each discrete state is weighted. For the baseline (Playstyle Distance), expected distributions according to observations are used. In contrast, for the metrics introduced after Section 3.2 in our paper, we apply a uniform distribution to weight each discrete state. This difference is not significant in stable or clear playstyles, but in unstable playstyles or with large action noises, the expected version sometimes yields better results.

Due to the unsuitability of OpenReview for long mathematical equations, we have added the extra metrics tested in the new experiments to Section A.5 of the updated manuscript for review.

RGSK 29 Trajectories

We used the 29 trajectories from the RGSK racing game, which were utilized in HSD training, as the basis for the playstyle identification task. Each trajectory is considered as a distinct playstyle, and we performed the same task as in our main paper for RGSK. The comparison sample size for each dataset is 1024 observation-action pairs.

These results indicate that the choice between uniform or expected versions significantly impacts accuracy, but the Jaccard Index is the most critical factor in this task.

Atari 80-20 Trajectories Identification

We used four levels of stochasticity from a single DRL model in our datasets, each consisting of 100 trajectories. We selected the first 20 trajectories of each level as the reference datasets for the four playstyles (stochasticity levels), and assessed the accuracy of predictions for the remaining 320 trajectories (4*80). Each trajectory is treated as a target dataset.

Asterix

Breakout

MsPacman

Pong

Qbert

Seaquest

SpaceInvaders

Summary of Atari Results

These results demonstrate that the Jaccard Index can sometimes have negative effects, highlighting the importance of high-quality discrete representation. We encourage the ICLR community to delve deeper into discrete representation for playstyle-related topics.