We thank the reviewer for their careful comments and critiques. We hope to address their primary concern through additional baseline experiments that were performed during the response period.

Specifically, we compare GAVEL against directly sampling from our fine-tuned language model and few-shot prompting with GPT-4o. For the direct sampling approach, we randomly select a game from the validation set and mutate it using the same selection and re-sampling process and repeat this process 4500 times (this is to mirror of the 500 generations of GAVEL, since in each generation we randomly select 3 games and perform 3 mutations on each). We then evaluate each of the generated games for fitness and determine the cell they would occupy based on their concepts. This allows us to construct a MAP-Elites archive (i.e. by keeping the most fit game in each cell) for the baseline in order to directly compare against GAVEL. We perform a similar process for GPT-4o, except we provide each of the 14 validation games as part of the prompt and ask the model to create a modification of one randomly-selected validation game. Due to constraints on budget we obtain only 900 samples from GPT-4o and thus compare to GAVEL after 100 generations. In a camera-ready version of the paper we will extend the GPT-4o baseline to the full 4500 samples and experiment with including additional context in the prompt.

The results of these baseline experiments are presented in the accompanying PDF, but the short summary is that both baseline methods fall far short of GAVEL in terms of QD score. Pure sampling appears to obtain dramatically fewer high-fitness samples both across the archive and in new cells. While GPT-4o produces a similar number of high-fitness samples in new cells it appears to produce dramatically fewer high-fitness samples overall. In both cases, it seems that GAVEL is doing a better job exploring the space of possible games. We will include these results in a camera-ready version of the paper.

In addition, we provide individual responses to the specific questions raised:

• Q1: How many games were filtered by each step of the process?

  • A: From the 1182 games in the full dataset, we first filter down to 1033 games in the “board” category. Removing the Mancala-style games brings the dataset to 825 games, and filtering by tokenized length brings us to our final 574 training games and 14 validation games.

• Q2: How is the concept vector defined for new games?

  • A: Ludii can automatically compute the concept vector for any game that successfully compiles: each feature is derived from the combination of “ludemes” (essentially, keywords) that are used in the game description.

• Q3: Doesn’t the QD score increase monotonically even without the offsetting of individual fitness values?

  • A: In most MAP-Elites implementations (and in ours), a sample can get added to a previously-empty archive cell regardless of its fitness score. This means that, in our case, games in novel cells with negative fitness scores can cause the QD score to decrease. We will make this distinction more clear in the camera-ready version of the paper.

We thank the reviewer for their careful comments and critiques and we hope to address their general concerns in our response. We also appreciate the pointers to additional related work and will include them in a camera-ready version of our paper.

While it is the case that the results we present are specific to games and the Ludii description language, we feel that our approach is general enough to apply to most domain-specific languages. DSLs are used in a variety of contexts and many of them are not well-represented in the training data of LLMs -- motivating a fine-tuning approach. Our fitness function is also general-purpose enough that it would work with minor modifications for other game-generation domains and the broad approach of a heuristic fitness function should in principle be applicable to an even wider range of domains. In this way, the broad GAVEL approach could be adapted for effective code synthesis in relatively data-sparse settings.

However, even if our method only works effectively for board game generation, this would in our opinion be both significant and general. Most of the world’s population play board games, at least occasionally, and the Ludii dataset contains more than a thousand games from all over the world. In the sense that our model is tasked with producing modifications to an existing game, it has demonstrated a broad and wide-ranging ability to do so. Further, the procedural generation of novel games presents an avenue for future research in open-ended learning.

Finally, we would also like to press the significance of the empirical results of our paper. We are the first to generate not one, but several new and interesting board games. The only directly comparable work is Cameron Browne’s Ludi system from 2009, but that operated in a significantly smaller search space consisting only of connection and line-completion games.

We thank the reviewer for their insightful comments. We acknowledge the reviewer’s points about the difficulty of evaluation and will include a more thorough discussion of these challenges in a camera-ready version.

We also provide individual responses to the specific questions raised:

• Q1: Is the used CodeLLaMA model the base-version or instructed version? Or which one performs better in your fine-tuning process?

  • A: We used the base version since we were fine-tuning on a single task. However, in some of our early experiments, we also fine-tuned instruct models to translate board game instructions from natural language to the Ludii DSL. While we eventually dropped that line of research due to the poor quality of the English descriptions accompanying Ludii games. We didn’t notice any significant difference in performance between the fine-tuned instruct and non-instruct versions of CodeLLaMA.

• Q2: The authors mention that the LLM would perfectly reproduce the missing section of code in the process of mutation. The proposed solution is to mutate a set of unseen games. It is surprised to me that the authors do not try to use top-k decoding or noisy decoding, since these methods are more intuitive and easy to me. Or these methods are not good in practice?

  • A: We agree that diversity-boosting strategies like top-k or noisy decoding are intuitive for increasing the novelty of mutations. However, our early experiments showed that these decoding strategies were ineffective at generating mutations that were both novel and valid. Given our relatively small pool of initial seed games, we found that simply excluding them from the training data was more effective. We will clarify this point in the camera-ready revision of the paper.

• Q3: I am not sure whether this work or the proposed method is greatly reliant on a high-level descriptive game language, or not? If we put this work in broader scenes, e.g. role-play games, do the principles and algorithms in the paper also work?

  • A: The approach we propose is applicable to most formal languages, including DSLs and general-purpose programming languages. The main constraint is that candidate games (or programs) need to be evaluated using a reward function. We achieve this by compiling the game descriptions, playing through them, and using heuristics to quantify the quality of the playthrough. Unlike strategy games, defining similar heuristics for games that feature a stronger human element, such as role-playing games, may be more challenging.

• Q4: It is not clear how to judge the correctness of the generated code, since the new games are represented in code (if I am wrong please correct me). Solely checking the compilation is not enough to determine if the code is right or not.

  • A: Indeed, compilability is only one of the metrics we consider when evaluating the quality or correctness of a game. Instead, we focus on evaluating the quality of playthroughs. We compile sample games into executable programs (they compile to Java bytecode), collect a series of playthroughs using self-play between MCTS agents, and evaluate each game’s playthroughs in aggregate using heuristics that quantify the game’s balance, decisiveness, completion, agency, and coverage. You can find more details about this process on pages 5 and 6.

• Q5: I hope the authors can discuss more recent work in AI generating games, e.g. Instruction-Driven Game Engines on Large Language Models (https://arxiv.org/abs/2404.00276). In this work, the authors automate the development process of poker games based on LLMs.

  • A: We thank the reviewer for pointing us towards this paper. While its approach is quite different from ours - they use LLMs as a game engine to predict next states, while we use them to define rulesets in a DSL - the authors' strong results are exciting. We will consider including it as related work in the camera-ready version of the paper.

We thank the reviewer for taking the time to carefully appraise our work. We hope we can adequately address their comments here.

First, while it is the case that our results are specific to games and the Ludii description language, we feel that our approach is general enough to apply to most domain-specific languages. DSLs are used in a variety of contexts and many of them are not well-represented in the training data of LLMs -- motivating a fine-tuning approach. Our fitness function is also general-purpose enough that it would work with minor modifications for other game-generation domains and the broad approach of a heuristic fitness function should in principle be applicable to an even wider range of domains.

Next, we take the reviewer’s point about the lack of comparative studies and attempt to address it through additional baseline experiments. Specifically, we compare GAVEL against directly sampling from our fine-tuned language model and few-shot prompting with GPT-4o. For the direct sampling approach, we randomly select a game from the validation set and mutate it using the same selection and re-sampling process and repeat this process 4500 times (this is to mirror of the 500 generations of GAVEL, since in each generation we randomly select 3 games and perform 3 mutations on each). We then evaluate each of the generated games for fitness and determine the cell they would occupy based on their concepts. This allows us to construct a MAP-Elites archive (i.e. keeping the most fit game in each cell) in order to directly compare against GAVEL. We perform a similar process for GPT-4o, except we provide each of the 14 validation games as part of the prompt and ask the model to create a modification of one randomly-selected validation game. Due to constraints on budget we obtain only 900 samples from GPT-4o and thus compare to GAVEL after 100 generations.

The results of these baseline experiments are presented in the accompanying PDF, but the short summary is that both baseline methods fall far short of GAVEL in terms of QD score. Pure sampling appears to obtain dramatically fewer high-fitness samples both across the archive and in new cells. While GPT-4o produces a similar number of high-fitness samples in new cells it appears to produce dramatically fewer overall. In both cases, it seems that GAVEL is doing a better job exploring the space of possible games. We will include these results in a camera-ready version of the paper.

Finally, we provide individual responses to the specific questions raised by the reviewer:

• Q1: Did you try not removing the Mancala-style games?

  • A: In initial experiments, our fitness evaluation had difficulty getting reasonable approximations for these Mancala-style games in the limited computational budget we provided. In addition, these games represent a fairly distinct cluster in terms of concepts of ludemes used and we omitted them in order to increase the relative “generality” of the training dataset, though future work could include them.

• Q2: Would CVT-MAP-Elites have worked in this case?

  • A: This is a very reasonable idea, and we did explore CVT-MAP-Elites in an initial experiment as well. Unfortunately running the CVT algorithm over the full concept vector space produced either a very sparse archive (when the number of cells was high) or a relatively collapsed archive (when the number of cells was small), which we attribute to the fact that the tessellation occurs uniformly over the space while the games are not distributed uniformly.

• Q3: Do you have ideas on how to characterize game diversity in other spaces than the BC space?

  • A: Capturing human notions of diversity in gameplay remains an open research question. Prior work does indicate that Ludii concepts capture certain high-level judgements of game genre (see citation [54] in our paper), but outside of this the gold standard would likely remain a human user study. We are in the process of organizing such a user study for future work / a camera-ready version of this paper.

• Q4: Why have you fine-tuned a smaller model instead of asking a large, capable instruction model to perform the task?

  • A: See baseline experiment above

• Q5: Could the various playability measures be collapsed to a single metric such as learnability, or a computational measure of interestingness?

  • A: While this idea is reasonable, we think that the main challenge would be in tractability. Using a measure of learnability as a fitness function would effectively add an inner loop to what is already the main bottleneck of the evolutionary algorithm. Perhaps such an approach would work in domains where generated game code could be compiled to something that runs on the GPU, but it would likely be infeasible for Ludii.

• Q6: Why do you think the UCB variant works less well?

  • A: One hypothesis is that it is very difficult to make any kind of improvement in the fitness landscape GAVEL operates over. The UCB MAP-Elites selector relies on the intuition that cells which produced successful mutations in the past are more likely to do so again. However, if improvement in any given cell is basically equally (un)likely, then the UCB algorithm would effectively waste time re-selecting previously successful cells instead of a more efficient uniform sampling.

• Q7: The limitation section discusses biasing mutation, do you have any idea on how this could work in practice?

  • A: In the context of Ludii, the most likely way to accomplish this would be with a heuristic-based method. It should in principle be possible to write a set of rules that link certain kinds of errors (e.g. unused components in the “equipment” section) with certain program regions (e.g. player actions in the “rules” section). Another possibility is an instruction-based mutation operator to which additional context (e.g. “this game has unused components”) could be provided.