Response Part 1/2:

We thank you for taking the time to provide a detailed review. We will attempt to address your questions (in addition to modifying our manuscript by adding emphasis on section 4.4 as described in our main response).

Your comments on Tables 1 & 2:

We thank you for the suggestion. We merged Tables 1 and 2 and agree that this has improved readability. We have left the error ranges in the appendix as they tend to clutter the tables and detract from the actual values. As for our speculation of why tree outperforms exp in hand and brain, we suspect that could potentially be due to the adversarial effect that tree can effect in HB. The exp agents in HB, with their horizon not extending beyond their hand’s moves, are the only agents actually able to reduce the loss of their hands. However, we expect that they will not be able to have an adversarial effect on the opponent hand, which might be quite important. That said, it is difficult to measure HB’s adversarial effect analogously because its interactions with the hands cannot be broken down into corrections and agreements due to it not outputting a full move.

Your comments on Section 4.2.2:

We have modified Section 4.2 to state the main takeaways more explicitly, and abstract the analysis to the appendix. The main takeaways from 4.2.2 and 4.2.3 are that all agents display some tricking effects (the expectors to the greatest extent). Expectors and tree exhibit a measurable helping effect. Finally, tree and attuned agents demonstrate a distributional effect. The analysis performed in 4.2.2 is less comprehensive (it neglects helping), but is significantly simpler than that of 4.2.3. The results of 4.2.2 and 4.2.3 show the same thing, except that 4.2.2 allows us to look at the interplay between the tricking effect and the distributional effect, whereas 4.2.3 gives us a clearer look at everything in isolation.

To clarify 4.2.2, the experiment demonstrates that exp is not dependent on a distributional effect as far as observed by the fact that its trickiness (which is high) is almost unchanged when measured on Leela’s distribution and its own distribution. For the other agents, we observe that trickiness is modulated by the distribution (the ability of focals to trick is amplified on the distribution of the focals compared to on leela's distribution), as well as a greater inherent tricking effect than leela even when standardized for distribution. The final observation was that if we had att and tree play on leela's distribution, they were less tricky than leela playing on their distributions, suggesting that their distributional effect is, in a sense, responsible for a large part (but not all) of their trickiness.

The "distributional effect" measured in 4.2.3 is slightly different than that in 4.2.2, it instead measures maia's performance when not preceded by any seniors (as opposed to how distributions modulate trickiness), and we see similarly that tree and att have a distributional effect there and exp does not.

Your comments on Section 4.4:

While we did not incorporate skill-uncertainty into our specific player compatibility experiments, we address the issue of skill-estimation in the supplement (1100 vs 1900). This does allow us to answer the question of how the results look when the amount of weakness is incorrectly estimated. Indeed, we find that seniors designed to play with stronger juniors fare better against weaker juniors than vice versa, and we suspect that overexploitation is punished whereas under-exploitation is not. This part was formerly in the appendix, and is now in the main body. We agree with you that testing att and tree is valuable on these agents, and we also understand the computational cost of the task. Accordingly, we add a previously omitted experiment where we run the other agents on these two players. We also implement your suggestion of reducing the 4.2.3 and 4.2.2 in favor of expanding this section notably, and having it play a more important role in the paper, which is our most major implemented revision after reading all reviews.

Your comments on the supplement:

Tying to our expansion of cross-style compatibility, and as mentioned in the main response, we also promote the cross-skill compatibility and Stockfish tests to the main body of the paper. We agree that they represent an important avenue for improvement and further work.

Thank you for your time and effort in reviewing our work, and for your support. Below we address the points you brought up in your review.

“Section 4.4 should be emphasized more from the beginning”
We agree that Section 4.4 was underemphasized in our original submission, and we will promote and expand it in our revision. We have uploaded a modified version of our manuscript demonstrating how we intend to clarify and expand it with analyses and text that we previously relegated to the appendix, as well as a small additional experiment. Thank you for this suggestion, we think this greatly helps the paper.

Relation to opponent-exploitation work in AI poker:

We agree that our work bears some similarities to the citations you listed, especially with regards to the use of a shallow MCTS to approximate the opponent agent at a particular moment of the game. One important difference is that the more sophisticated versions of adversarial MCTS deployed in these papers to pinpoint weaknesses in strong opponents are not necessary in ours to glean a noticeably advantage against the mixed opponent team (and in fact we notice that even 1-ply horizon EXP agents are still able to obtain good scores in the Appendix.) We will include the citations you listed and clarify this point in the main text.

What is the relationship between this work and that of the exploitability and adversarial attacks fields?

Thank you for pointing out that our work is related to the exploitability and adversarial attacks fields; we have updated the manuscript with citations to this literature. One main difference between this work and ours is that in our problem formulation, beyond adversarial play, we also have a junior partner agent on our side whom we wish to influence positively (or at the very least, protect from our adversaries), and we are accordingly not in full control of even our own team’s outputs. Another difference is in scope; these works develop technical methods to identify and exploit suboptimal play in sophisticated agents, whereas our agents sometimes learn simple ways to exploit a relatively simple agent (Maia). We find that it could be conceptually interesting to formulate the opponent team in a similar fashion to that done in [3], with the opponent being a mixed strategy that samples from a supposedly “optimal” strategy (that of the senior) and occasionally “gifts” when the junior plays.

We thank you for taking the time to review our paper. We agree that a small human trial would be of interest in extending our concept of skill-compatibility to human-compatibility. And alongside further investigation of the concepts of measuring how skill-compatibility holds up under different styles and skill gaps, determining how skill-compatibility translates to actual human-compatibility is an open question worthy of further investigation. If a researcher wished to test our central result exemplified in table 1, we have uploaded the necessary weights and code onto our github for that purpose.

In response to the specific questions: Tables 1-3 could be explained more clearly. Please consider bolding best results.

Thank you for the suggestion. We have bolded the best results in Tables 1 and 2 (now merged for clarity) and rewrote the explanation for Table 3 for clarity.

Why is the EXP method N/A in table two?

The exp agent designed for HB only selects a piece based on its anticipation of maia’s actual move, and therefore is unable to actually play chess. The other agents are adapted to HB by simply masking the second part of the move, meaning Nf6 simply gets output as N in HB, with Nf6 still technically being there. exp simply outputs N, we don’t have anyway to get Nf6.

Do you have any intuition as to why EXP performs best overall? Why does the paper not claim this as the best method if it has the highest winrate?

Our intuitive guess as to why EXP performs well, especially in STT, is that accurately modeling the game for a few turns is more valuable than a long-term approximation in a highly noisy environment where the junior’s blunders are likely to throw the game off into different paths. The MCTS employed by other seniors refines the search assuming the opponent and the self are using a strong policy, but the presence of the weaker juniors means this behavior may not be not entirely accurate.

The results are explained well after reading through the experiments section many times.

In accordance with your suggestion, we have modified our results section to more clearly state our results immediately after the analysis.

Thank you for taking the time to review our paper. Below we address the points you brought up in your review.

"Since you use the weaker chess engine as a subroutine in search, it isn’t clear how you could make a version with humans since they can’t communicate at test time."

We apologize for the insufficient clarity on this point, which we have addressed in our updated manuscript. The attuned agents do not use the juniors as subroutines; they only train with them prior to testing. Thus one contribution of ours toward broader skill- and human-compatible collaboration is developing an agent that doesn’t need to communicate at test time. In response to your feedback and to that of the other reviewers, we have emphasized Section 4.4 and the points made within it to clarify the fact that our models do not need exact models of their opponents at either training or testing times. The updated section explicitly addresses your point concerning the ability of our agents to work without having an exact model of their juniors. Our results in this section show that even in an environment where we cannot communicate with the juniors and don’t have a correct model of them, generic approximations still perform well (e.g. one experiment takes expector seniors that use generic maia models as subroutines, and they are able to beat leela when they matched with specific player models that they do not have access to).

“These methods seem limited to chess. It would be more interesting to have methods that would work for a variety of cooperative or cooperative/competitive games”

We start by noting that our methods could extend to other two-player zero-sum game games of perfect information, but we completely agree that developing methods for a broader spectrum of games, including cooperative games, is a worthy goal. Our belief is that doing so is a larger research agenda than one paper, so our approach here is to make tangible progress starting with chess. Like DeepMind did by starting with AlphaGo and then generalizing from there, we hope to start with the agents presented in this paper for chess and generalize to a broader set of domains.

Questions How can the methodologies be adapted for more dynamic, less rule-bound environments outside of chess?

This is a key question for the line of work that we hope this paper helps kickstart. Similar to how AlphaZero was a separate follow-up to the already-impressive AlphaGo work, adapting our methodology for chess to more dynamic, less rule-bound environments is its own worthy goal.

Would the proposed techniques be effective in real-time scenarios where the lower-skilled entity is a human?

We believe the answer is yes. Our closest approximation of this is playing versus specific human models partnered with generic seniors. In this setup, our agents do not have access to exact models of their junior partners (only to generic approximations). Although these player-specific models are not identical to individual human players, they are high-fidelity models of them, and they perform well. Real-time constraints are not an issue here, as moves are generated in the order of seconds by our agents.

How does the proposed skill-compatible AI adapt to the continuous learning and evolving capabilities of lower-skilled agents?

In this work we considered static agents, but as shown in section 4.4 the resulting models generalize to different agents (models of individual human chess players). As such, we expect that our methods could be adapted to work with continuously changing agents via periodic updates to the model.

In real-world applications, how can the balance between raw AI performance and skill-compatibility be optimized without compromising on crucial tasks?

This is a question that depends significantly on the domain. In chess AI performance is significantly greater than human so we are happy to trade a large amount of performance for skill-compatibility. If these methods were deployed in a real-world situation both the senior and junior agents would be required to act, e.g., the senior has an unpredictable delay between uses (an overtaxed server). In a situation like this obtaining compatibility would be of utmost importance as the junior agent might at any time have to complete the task with no support.

Could the techniques be integrated into current AI frameworks for instant benefits, or would they require a complete overhaul of the system?

In short, yes this can be integrated. Our methods require a good (but not perfect) model of humans to use for search/training. These models have become more and more common as training foundation models/semi-supervised training is very popular. If this model is available then our training loop can be easily integrated into an existing RL search/training framework. In fact, our own code was integrated into a Tensorflow MCTS search system without too much effort.