In-Context Learning for Games

Thank you for your valuable feedback. Below is our response to your question:

---

Weakness 1

• Algorithm 3 integrates opponent exploitation within the PSRO framework. In this design, opponent exploitation replaces the best response oracle in the vanilla PSRO algorithm, offering a more efficient alternative by reducing computational overhead.

• The key advantage of Algorithm 3 is its ability to function without requiring parameter updates. In vanilla PSRO, the best response oracle involves training BR policies, which can be computationally expensive. In contrast, Algorithm 3 utilizes in-context learning to derive the BR policy directly, eliminating the need for parameter updates and significantly enhancing efficiency.

---

Weakness 2

• Indeed, integrating ICE into RL could enhance opponent exploitation. However, a traditional RL algorithm is typically designed to optimize the utility of a single player given a fixed opponent strategy. In this paper, our goal is to train a model capable of playing as any player in the game, enabling it to exploit its opponent while also contributing to equilibrium strategy computation.

• If ICE were integrated into RL, the RL framework would be limited to optimizing for a single player’s perspective, reducing its ability to generalize across different player roles in the game. Therefore, this method cannot solve our problem.

---

Weakness 3

• We agree that comparing a pre-trained model (ICE) with a non-pre-trained model (PPO) might seem to give ICE an inherent advantage at earlier time steps. However, the purpose of this comparison is to demonstrate the ability of ICE to adapt efficiently to different opponent strategies using in-context learning without parameter updates, as opposed to PPO, which requires continuous parameter optimization.

• The performance drop observed in Figure 4 and the slightly worse performance of ICE in out-of-distribution (OOD) settings are important findings. These results indicate that while ICE excels in in-distribution settings, its adaptability to OOD opponents and long-term performance can be further improved.

---

Weaknesses 4, 5, and Question 3

• In the paper, we have conducted ablation studies on different context lengths (RQ6) and different player positions (Appendix C). All these results are evaluated in both the in-distribution and out-of-distribution testbeds. Additionally, Figure 8 shows that ICE outperforms multi-task pre-training with fine-tuning frameworks in OOD scenarios in the Leduc poker game.

• For parameter settings, we have provided the necessary parameter table in Appendix B. In future revisions, we will consider including key parameters directly in the main paper to improve accessibility and clarity.

---

Question 1

• Thanks for pointing out. These terms refer to the same concept. We have unified the terminology throughout the paper for clarity in the revision. In Figure 9, it should be “w/o” instead of “w/” and we have corrected it in the revision.

---

Question 2

• ICE benefits from pre-designed diverse opponent strategies to train a transformer-based model capable of in-context learning. This design provides several advantages, including rapid adaptability and parameter-free inference during deployment. Although these benefits come at the cost of requiring a curated training dataset, which may involve computational overhead during pre-training, ICE has more advantages than RL and equilibrium-finding algorithms.

• RL approaches, such as PPO, optimize policies online and do not require pre-designed opponent datasets. However, they often involve continuous parameter updates and can incur significant computational costs during deployment, especially when adapting to new opponents. Contrast this with ICE, ICE shifts computational effort to the pre-training phase, enabling faster adaptation during inference.

• PSRO typically requires iterative computation of best responses and meta-strategies, which can be computationally expensive, especially in large games or with complex opponent pools. In contrast, ICE uses in-context learning to approximate best responses without requiring parameter updates, potentially reducing application costs.

---

We hope our response can address your concerns, and we deeply appreciate that if you could reconsider the evaluation of our paper.

Dear Reviewer wMz9,

Thank you once again for your valuable review.

As the author-reviewer discussion period is nearing its end, we would like to kindly ask if our responses have addressed your concerns. We have put a lot of effort into preparing detailed responses to your questions. We are eager to address any additional feedback or unresolved concerns you may have before the discussion period concludes.

We look forward to hearing from you. Thank you for your time and consideration.

Sincerely,

Authors of Submission 8911

Dear Reviewer wMz9,

Thank you for your time, effort, and thoughtful feedback in reviewing our work. As the author-reviewer discussion period approaches its conclusion (11:59pm AoE on 2nd Dec), we sincerely hope that our responses have addressed your concerns.

If our responses have adequately resolved your concerns, we would be deeply grateful if you could reconsider your evaluation of our submission. However, if there are any remaining issues or uncertainties that you feel we have not sufficiently addressed, we kindly ask for further clarification or suggestions. We are eager to make additional improvements before the discussion period ends to ensure our work meets your expectations.

Thank you again for your valuable feedback and for helping us improve our paper.

Sincerely,

Authors of Submission 8911

---

Reply to Q12

• The purpose of mixing in data from random strategies is to enhance the diversity of the training set, ensuring that the model can generalize to a broader range of potential opponent behaviors, including less structured or irrational strategies. We acknowledge that introducing random strategies at regular intervals may create a non-monotonic (jagged) difficulty sequence. However, our intention is not to strictly follow a monotonically increasing difficulty but rather to strike a balance between exposing the model to diverse behaviors and progressing toward harder tasks.

---

Reply to Q13

• Although current self-play does not guarantee convergence, it is often used as a framework to approximate equilibrium strategies due to its simplicity and practicality. To support this, we have added experimental results for Leduc Poker and Goofspiel, demonstrating that self-play can perform well, particularly in scenarios involving pure equilibrium strategies.

---

Reply to Q14

• Here, the “out-of-distribution” testbed refers to opponent strategies that are not included in the training dataset. To create the out-of-distribution testbed, we rerun the random generation method to produce a new set of random opponent strategies.

---

Reply to Q15

• Here, we use the CFR algorithm, as implemented in OpenSpiel, to compute the NE strategy for the three-player Kuhn Poker. While computing NEs in multi-player games is generally very challenging, three-player Kuhn Poker is relatively simple, and the CFR algorithm can effectively compute an approximate NE strategy in this case.

---

Reply to Q16

• Here, “game iterations” refers to the number of games played during the evaluation process. Each game iteration represents a complete playthrough of the game, and the x-axis tracks the cumulative number of such iterations.

---

Reply to Q17

• Indeed, ICE leverages cross-episode trajectories as part of its in-context learning process. However, while the PPO policy is not explicitly conditioned on cross-episode trajectories during inference, it does use these trajectories as training data to update the policy.

• Additionally, each in-context episode is independent of all other in-context episodes, as each episode represents a single game process, and every game process is independent of one another. Therefore, we evaluate both algorithms based on the number of game iterations.

---

We hope our response can address your concerns, and we deeply appreciate that if you could reconsider the evaluation of our paper.

Thank you for your valuable feedback. Below is our response to your question:

---

Reply to Q1

• We appreciate your thoughtful analysis of both the motivation behind our approach and the clarity of our experimental setup.

• Unlike traditional uses of transformers for in-context learning, we do not leverage pre-trained general-domain knowledge. Instead, we generate domain-specific data to train a specialist transformer model capable of leveraging in-context learning. This focus on domain-specific training enables the model to address the unique challenges of multiplayer games, such as strategic reasoning and opponent exploitation, while remaining parameter-free during inference.

• The key motivation for leveraging transformers lies in their ability to process long trajectories and encode complex strategic patterns within a single framework. While prior work has achieved notable results using other architectures, our results suggest that transformers, with carefully constructed domain-specific datasets, offer a flexible and scalable approach to generalization across diverse opponents. This is particularly relevant for equilibrium computation tasks, where long-term dependencies between player strategies must be modeled effectively.

• As introduced in Experimental Setup. In the in-distribution testbed, the opponent policies are sampled from the same set of strategies used during the training phase. For the in-distribution testbed, we sampled about 30 opponent policies.

• We use the online setting for evaluating our algorithm. Therefore, for ICE(AD), the in-context examples are generated by trajectories played with the given opponent previously, which means the context would be empty at the beginning.

---

Reply to Q2

• Here, we intend to convey that NE computation can be computationally intensive. We use PSRO as an example to illustrate why this process often involves significant computational overhead. We have revised this description in the revision.

---

Reply to Q3

• Indeed, NE is difficult to compute for games with more than two players. Additionally, in multi-player games, other equilibrium strategies may be more suitable solutions than NE, depending on the specific context and objectives of the game.

---

Reply to Q4

• Thank you for your suggestion. We have removed this figure in the revision.

---

Reply to Q5

• Thank you for pointing this out. We have revised it in the revision.

---

Reply to Q6

• Thank you for highlighting the ambiguity in L76-77. It should be “the trained model can be used to compute NE with equilibrium finding algorithms”. We have revised it in the revision.

---

Reply to Q7

• Thank you for your valuable feedback. We appreciate your suggestion to reference Simplex-NeuPL and have incorporated it into our discussion to provide a more comprehensive context for our motivations. In the revision, we have replaced Figure 2 with a payoff table for clarity and included Simplex-NeuPL to strengthen the connection to related work addressing similar problem settings.

---

Reply to Q8

• Thank you for pointing this out. We have revised it in the revision.

---

Reply to Q9

• There is a misdescription in the original paper regarding the opponent strategies generated by the learning-based method and we have revised it in the revision. These strategies are the opponent’s final strategies observed during the learning process. For the PSRO algorithm, the meta-strategy over all strategies represents the final result.

• In our experiments, we use the CFR algorithm, where the opponent’s final strategies at each iteration are its average strategies among all previous strategies. As the algorithm progresses, these average strategies gradually converge toward the NE strategy. Therefore $D_{l1}$ and $D_{lb}$ are respectively the easiest and hardest strategies to exploit.

---

Reply to Q10

• The training dataset for DPT requires an optimal strategy to provide the optimal action as the label for any query state. To collect in-context data (i.e., trajectories), we use PPO to learn the optimal strategy against a fixed opponent, which is then used to generate the training trajectories. Consequently, the trained optimal strategy obtained via PPO serves as the optimal strategy required for constructing the training dataset for DPT.

---

Reply to Q11

• Yes, the gap refers to NashConv. While computing the NashConv for a single strategy is straightforward in these games, the presence of numerous opponent strategies makes the process time-consuming due to the need to compute it for all opponent strategies.

Dear Reviewer XVLk,

Thank you once again for your valuable review.

As the author-reviewer discussion period is nearing its end, we would like to kindly ask if our responses have addressed your concerns. We have put a lot of effort into preparing detailed responses to your questions. We are eager to address any additional feedback or unresolved concerns you may have before the discussion period concludes.

We look forward to hearing from you. Thank you for your time and consideration.

Sincerely,

Authors of Submission 8911

Dear Reviewer XVLk,

Thank you for your time, effort, and thoughtful feedback in reviewing our work. As the author-reviewer discussion period approaches its conclusion (11:59pm AoE on 2nd Dec), we sincerely hope that our detailed, point-by-point responses have addressed your concerns.

We have dedicated significant effort to revising our paper and preparing thorough answers to your questions. If our responses have adequately resolved your concerns, we would be deeply grateful if you could reconsider your evaluation of our submission. However, if there are any remaining issues or uncertainties that you feel we have not sufficiently addressed, we kindly ask for further clarification or suggestions. We are eager to make additional improvements before the discussion period ends to ensure our work meets your expectations.

Thank you again for your valuable feedback and for helping us improve our paper.

Sincerely,

Authors of Submission 8911

Thank you for your valuable feedback. Below is our response to your question:

---

[Weakness for RQ3]

• We agree that the current results for RQ3 are mixed, and presenting them as an affirmative answer oversells the findings.

• In the revision, we have added experiments on Leduc Poker and Goofspiel (Figure 5) and clearly described the mixed nature of the results. We emphasize that while ICE demonstrates progress toward computing NE without parameter updates, the exploitability remains relatively high by the end of the experiment for games with high dynamics. However, in simpler games, such as Goofspiel with 3 cards, our ICE achieves a low exploitability. We believe this is due to the existence of a pure strategy equilibrium in this particular game.

• Additionally, we have reframed the discussion to highlight these results as a promising first step rather than a definitive solution, while acknowledging the gap that remains to be addressed in future work.

---

[Opponent Strategy Generation]: The setup of the Opponent Strategies Generation is ill-informed

• There is a misdescription in the original paper regarding the opponent strategies generated by the learning-based method and we have revised it in the revision. These strategies are the opponent’s final strategies observed during the learning process. For the PSRO algorithm, the meta-strategy over all strategies represents the final result.

• In our experiments, we use the CFR algorithm, where the opponent’s final strategies at each iteration are its average strategies among all previous strategies. As the algorithm progresses, these average strategies gradually converge toward the NE strategy. Therefore $D_{l1}$ and $D_{lb}$ are respectively the easiest and hardest strategies to exploit.

---

[Experimental Results in 5.2]: The BR (red line) is a fixed policy (not changing as the number of game iterations increases), right?

• BR (red line) and NE (purple line) are fixed policies. As our results are obtained from simulations, we plotted the curve to illustrate that the estimation for these policies converges to a fixed value as the number of simulation iterations increases. We appreciate your suggestion, and we will consider representing these values as flat horizontal lines in revisions to make the results clearer and emphasize the fixed nature of these policies.

---

[OOD setting]: It should be pointed out that ICE does not seem to converge to the BR in the OOD setting.

• Thanks for pointing this out. We have revised it in our revision.

---

Q1: Did you evaluate against just a single opponent per chart?

• The experimental results are based on evaluations against a set of opponents, as outlined in the Experimental Setup. Specifically, we used 30 opponents for the in-distribution testbed and 20 opponents for the out-of-distribution testbed. Since these opponents exhibit significant variability and the utility against them has a high variance—ranging from the lowest to the highest returns—we opted to plot only the average results in the charts without including error bars.

---

Q2: Is it not expected that the average return of NE vs. NE in Kuhn poker is not 0?

• Thank you for the comment. The Nash value for Kuhn Poker is indeed 1/18. Here, we intend to highlight that the sum of the average returns of NE and BR strategies for all players is not exactly zero due to the approximation of NE in our experiments. This has been clarified in the revised manuscript.

---

We hope our response can address your concerns, and we deeply appreciate that if you could reconsider the evaluation of our paper.

---

Q5: Ablation in terms of which part of the CL is used

• We have updated Figure 7 in the revision to include two additional ablation results for a fair comparison. All comparisons are conducted using the same set of opponent strategies. Specifically, we compare our method with two training strategies: Training first on random opponents, followed by learning-based opponents (“Random First”), and Training first on learning-based opponents, followed by random opponents (“Learned First”).

• As shown in Figure 7, ICE, Random First, and Learned First all outperform ICE without CL. Interestingly, ICE, Random First, and Learned First achieve similar overall performance, which may be attributed to the inherent curriculum-like nature of learning-based opponents.

• However, we observe that Learned First performs better in in-distribution cases, likely due to the stronger initial focus on structured strategies and Random First performs better in out-of-distribution cases, benefiting from early exposure to diverse random strategies. Our ICE method demonstrates more stable performance across both in-distribution and out-of-distribution settings. This indicates that integrating random opponents into training with learning-based opponents enhances the robustness of the training process, leading to a more balanced and adaptive model.

---

Q6: About the third part of the desiderata

• The third desideratum refers to the ability of the pre-trained model to assist in computing Nash Equilibrium (NE) strategies, without requiring additional parameter updates. Specifically, the model leverages its in-context learning capabilities to represent and adapt strategies dynamically by modifying the context, rather than altering its internal parameters.

• The motivating example highlights the limitations of NE strategies in certain scenarios, particularly against opponents who deviate from equilibrium behavior. It demonstrates the need for a model that can exploit such deviations to maximize utility, rather than rigidly adhering to NE strategies.

• The third desideratum and the motivating example serve complementary purposes. The motivating example illustrates the flexibility of the proposed framework in exploiting non-equilibrium opponents, while the third desideratum ensures that the same model can also contribute to NE computation when required.

---

Q7: In the card games, what does it mean for the framework to handle different players? Are the different players just defined by order of which of the players plays first?

• Note that these card games are not symmetric games. “different players” refers to the distinct roles that each player assumes in the game, not just the order in which they play. For example, the information available to a player at any point in the game varies depending on their role and each player may face different sets of possible actions depending on the game state and their position in the game.

• Optimal strategies for one player often depend on the strategies of the other players, which makes playing as different players a distinct task. Our framework is designed to train a single model that can play as any player in the game. This means the model can adapt its behavior based on the player role it is assigned and the corresponding context. Therefore, handling different players in our framework involves adapting to the unique roles and decision-making scenarios of each player, which go beyond just the order of play.

---

Q8: For the out-of-distribution testbed, where were the randomly sampled opponent strategies from? Were they just random strategies?

• To generate the out-of-distribution testbed, we rerun the random generation method to generate these random opponents not just random strategies.

---

Q9: In Fig. 10, what is the difference between the two subplots in (a) and (b)?

• The two subplots in (a) and (b) represent results for in-distribution and out-of-distribution test cases, respectively. We have clarified this distinction in the revised manuscript to ensure a better understanding.

---

We hope our response can address your concerns, and we deeply appreciate that if you could reconsider the evaluation of our paper.

Thank you for your valuable feedback. Below is our response to your question:

---

Q1: Can the authors demonstrate the diversity of the data that is collected?

• Our generated opponent strategies include both random strategies and those derived from the equilibrium finding algorithm. As outlined in the paper, the random generation step serves to create a baseline of varied and unpredictable strategies, mimicking scenarios where opponents may act irrationally. These random strategies inherently contribute to the overall diversity of the dataset.

• We acknowledge the limitations in the diversity of strategies generated by the PSRO algorithm, as highlighted in the literature. In the paper, we use the PSRO algorithm as an illustrative example to demonstrate how opponent strategies can be recorded (i.e., the meta-strategy over strategy pools) during the equilibrium learning process, capturing different levels of opponent capabilities. In our experiments, we apply the CFR algorithm to generate these opponents. Specifically, the average strategies generated among all previous strategies during each iteration of CFR are added to the opponent strategy pool, ensuring a broad representation of different skill levels and behavior types. These strategies are designed to reflect a spectrum of opponent capabilities, with their varying proximity to the NE strategy. We also acknowledge a misdescription in the original paper and have corrected it in the revised version.

• By combining these two types of opponent strategies—randomly generated and learning-based—we aim to construct a dataset that is diverse both in behavioral patterns and skill levels.

---

Q2: Could the authors maybe present a stronger argument for why the order of the curriculum they proposed is good for this framework?

• There is a misdescription in the original paper regarding the opponent strategies generated by the learning-based method and we have revised it in the revision. These strategies are the opponent's final strategies observed during the learning process. For the PSRO algorithm, the meta-strategy over all strategies represents the final result.

• In our experiments, we use the CFR algorithm, where the opponent’s final strategies at each iteration are its average strategy among all previous strategies. As the algorithm progresses, these average strategies gradually converge toward the NE strategy. If we consider the NE strategy as the most challenging opponent to exploit, the sequential order of these strategies naturally forms a suitable curriculum for training.

---

Q3: To me personally, it feels counterproductive to the learning process to include the random strategies in the curriculum process at all

• As we introduced, the random generation of opponent strategies is intended to enhance the overall diversity of the opponent strategy set. While learning-based strategies provide structured and progressively refined behaviors, they may lack sufficient diversity to fully train a robust and adaptable model. By incorporating random strategies into the curriculum, we aim to expose the model to a broader spectrum of behaviors, including unpredictable and irrational actions, thereby enhancing its robustness.

• To address the concern about catastrophic forgetting, our curriculum learning framework includes a revisiting mechanism. This mechanism allows the model to periodically retrain on previously encountered strategies, ensuring that knowledge gained from earlier tasks is retained while the model learns to adapt to new strategies. This approach mitigates the potential for random strategies to disrupt the learning process, maintaining stability and preventing the model from overfitting to specific opponent behaviors.

---

Q4: Why this is an approach that could be ultimately be better even with the potentially large amounts of pre-training required?

• Parameter-Free. While it may appear resource-intensive, once pre-trained, the model can adapt to new scenarios or opponents through in-context examples without requiring any further updates to its parameters. This is particularly advantageous in real-world applications where retraining or fine-tuning may be infeasible due to time, computational constraints, or the need for rapid deployment.

• Generalizability. The model trained through our approach can handle a wide variety of opponents, ranging from random strategies to those close to Nash Equilibrium, providing robustness and flexibility in unpredictable environments.

• Efficiency During Inference. While the pre-training phase may involve significant computation, the inference process is lightweight, as it leverages a pre-trained model and in-context learning to dynamically adapt. This makes it suitable for scenarios where inference needs to be conducted efficiently and repeatedly.

Dear Reviewer Xvmq,

Thank you once again for your valuable review.

As the author-reviewer discussion period is nearing its end, we would like to kindly ask if our responses have addressed your concerns. We have put a lot of effort into preparing detailed responses to your questions. We are eager to address any additional feedback or unresolved concerns you may have before the discussion period concludes.

We look forward to hearing from you. Thank you for your time and consideration.

Sincerely,

Authors of Submission 8911