Thank you for the detailed and thoughtful feedback. We’ve clarified the setup, improved the analysis, and incorporated the suggested related work. We appreciate your input and believe it has strengthened the paper.

Strengths

We appreciate the positive assessment of the motivation, experimental design, and broader implications of our work. We're encouraged that the framing aligns with future research directions.

Weaknesses

Weakness 1A: Train/Test Partner Overlap

Thank you for the great point. While some individual speed values overlapped between training and test, the sets were disjoint in terms of speed pairs (e.g. (2,3)), as all speed pairs used during training were excluded from the test set. This ensured that the ego agent was never paired at test time with a teammate policy it had encountered during training. We apologise for the confusion, and will update the manuscript to make this clearer.

Weakness 1B: Online Adaptation and Training Set-up

This is also a good point. During evaluation, to simplify the analysis, we always switched the teammate properties in a single direction and at t=300. During training, however, both the timing and the nature of the switch were randomised: the ego agent encountered teammates whose properties switched in both directions (i.e. faster at task 1 → faster at task 2, or vice versa), and t_switch was sampled uniformly between 25% and 75% of the episode length. Additionally, we performed the switch only in 50% of episodes. This was done exactly to mitigate the concern that the model might memorise a fixed timing or pattern, rather than truly adapting to the change online. We will also update the manuscript to clarify this.

Weakness 2: Partner Modelling and Decoder Analysis

Thank you for the feedback; we acknowledge that our wording could be clearer and more specific here. While the baseline models do contain some partner-related information, as evidenced by above-chance decoding performance, the influencing condition shows a substantially stronger emergence of partner modelling. We will update the manuscript to clarify this.

Comments

Comment 1: Additional Analysis

This is a good point. We chose the UMAP projections as a way to produce clear visual evidence of our findings, and the linear probe analysis for its conceptual simplicity. That being said, we agree that additional analyses would help strengthen our conclusions. The idea of looking at per-unit correlations to better understand at what ‘level’ the representations are being learned is appealing, and has a nice mechanistic interpretability-like flavour. We also thank the reviewer for pointing us to DSA, which we were not previously familiar with. We will make sure to incorporate at least one additional analysis approach into the revised manuscript.

Comment 2: Performance Drop on fivebyfive_v1 Configuration

This is a good observation - we believe the exception in fivebyfive_v1 highlights that partner modelling is environment dependent. In fivebyfive_v1, spatial coordination is less critical than in cramped_room variants (due to the simpler and more open geography), making feedforward models more competitive. Additionally, these plotted results are from the fully observable setting, where agents can often infer their partner’s behaviour from the current state alone — reducing the benefit of temporal modelling. In contrast, the performance gap to MLPs is larger in the partial and blind settings, where temporal reasoning becomes more important. Another possible factor is that agents are evaluated with novel partners. RNNs may overfit to training partners, whereas MLPs, lacking memory, may generalise better when spatial context is sufficient.

Comment 3: Related Work

Thank you for sharing these relevant and insightful papers. We’ll incorporate all of the papers into the literature review to make it more comprehensive and to better situate our contributions within the broader context. It was nice to read the papers - they are relevant and great suggestions.

Thank you for your thoughtful feedback on our paper. We appreciate your recognition of the paper’s clarity, the use of analytical tools like UMAP and linear probing, and the careful design of controlled experiments. We’ve addressed all of your points, including obtaining preliminary results in a second environment to strengthen the empirical validation of our claims. We have incorporated, or are incorporating, your questions and suggestions into the revised manuscript, which we believe is stronger as a result and hopefully addresses your concerns.

Questions

Q1 Why is an RNN used for attention…

Thank you for bringing this work to our attention. Long et al.’s paper shares a similar motivation (representation and theory of mind in multi-agent settings), but their approach is different. While we focus on implicit partner modelling, they train agents explicitly to predict/infer each other’s attention. We will update the related work section to clarify this distinction.

Regarding the broader point about RNNs versus attention/transformers, in this work, we chose an RNN because it is a standard architecture for memory-based agents in MARL (Foerster et al., 2016), and offers simple, interpretable internal dynamics. Our aim was to explore whether structured representations emerge without strong architectural inductive biases, and the RNN is well suited to that goal.

While transformers could also be applied here, they typically require larger datasets or pretraining to be effective (Chen et al., 2021), which makes it less suitable for our setting. Additionally, to our knowledge, transformers are mainly used in offline RL (Chen et al., 2021). That said, using transformers to probe agent representations (particularly via attention) would be a promising direction for future work, and we now discuss this explicitly in the revised manuscript.

Q2. How does the ego agent collaborate with humans?

We do not evaluate collaboration with human partners, but this is an important direction for future work. A key challenge is mutual adaptation: unlike fixed-policy partners, humans adjust in response to the agent, changing the learning dynamics. Understanding how internal representations evolve under such bidirectional influence would be especially valuable. We have updated the manuscript to highlight this as a promising area for future work.

Q3. Scaling to more agents

This is an interesting direction for future work, and we have updated the manuscript to discuss it more explicitly. We see no reason why the same kind of implicit representation would not scale to more than one teammate, provided the environment imposes sufficient pressure—i.e. all teammates’ behaviour meaningfully affects task success. That said, inference becomes more demanding with additional agents, as more evidence is required to identify their individual properties. The ego agent may adopt useful approximations, such as modelling the average behaviour of its teammates or inferring which teammate is best suited to each subtask.

Q4. If the number of patterns (?) is very large…

Thanks for the question. In our experiments, the ego agent is exposed to over 50 unique partner pairings, with most variation along the axes of speed and randomness. As partner diversity increases across other behavioural dimensions, we would expect the agent’s internal representations to expand accordingly—encoding not just speed, but also traits like reliability, preferences, or coordination style. Exploring this broader space is an interesting next step. We will update the manuscript to highlight this as an interesting future direction.

Q5. In the partially observable environment…

In a partially observable environment, the ego agent can't always see its partner and must rely on memory to infer their behaviour. It can still model the partner, but this takes longer, as observable evidence of the partner’s attributes is more sparse. With a radius of 1 (a 3×3 grid centred on the agent), the agent gets some local observations and can adapt fairly quickly. In the blind case, with no visual input, adaptation is slower (but does still occur). The RNN was crucial here, allowing the agent to use past experience to model both the partner and the environment state—for example, it may try to pick up soup, notice no onion arrives, infer the partner is incompetent, and adjust its strategy accordingly.

Limitations / weaknesses

Do the findings hold true in other environments and under different settings?

To address this point and test the robustness of our findings, we have conducted a preliminary experiment in an additional environment; and we will expand upon these results in the revised manuscript.

We agree that demonstrating generalisation beyond Overcooked-AI is a critical next step. Our goal in this paper was to establish the emergence of partner modelling under minimal assumptions, using a rich cooperative domain that allows for clear analysis of agent memory and adaptation. That said, we expect the underlying mechanism — implicit modelling driven by reward and influence — to be broadly applicable. We believe similar dynamics are likely to emerge in other cooperative settings where agents must adapt to diverse partners, especially under partial observability.

To validate this, we conducted preliminary experiments in a second environment: a modified version of CoinGame (from the JaxMARL suite). In our version of the environment, the ego agent must collaborate with a teammate to collect red and blue coins in a small gridworld, and is rewarded based on the total number of coins collected (i.e. fully cooperative reward function). While the ego agent can collect both coin colours, the teammate is only able to collect either red or blue – and, analogously to our overcooked setup, the ego agent has an additional action allowing them to control which colour coins the teammate attempts to collect. Consistent with our results on overcooked, we find the following:

• RNN agent with influence + trained alongside both teammate types outperforms RNN without influence, RNN trained only alongside a single partner, and MLP (in terms of total coins collected) (Table 1)
• In evaluation episodes with the influence RNN agent, the teammate spends close to 80% of their time pursuing the “correct” coin colour (relative to a ~50% baseline) (Table 1)
• Teammate types are more (linearly) recoverable from the hidden states of RNN agents with influence than those without (Table 2)

Table 1 (mean/std taken over 5 training seeds and 1000 eval eps):

Table 2 (mean/std taken over 5 training seeds and 1000 eval eps):

These results suggest that the phenomenon is not limited to Overcooked, but instead reflects more general properties of cooperative RL under the right environment pressures.

References

Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., & Abbeel, P. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. Advances in Neural Information Processing Systems, 34, 15084–15097.

Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016). Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Advances in Neural Information Processing Systems, 29.

Thank you for the thoughtful and constructive feedback. We appreciate the recognition of our experimental design, evaluation, and key findings on implicit partner modelling. We have addressed all of the points raised, including conducting preliminary experiments in a second environment, and believe the paper is stronger as a result.

Weaknesses

1. Generalisation beyond Overcooked-AI

We agree that demonstrating generalisation beyond Overcooked-AI is important. Our goal was to show that partner modelling can emerge under minimal assumptions in a rich cooperative domain. We expect the same reward- and influence-driven dynamics to arise in other settings where agents adapt to diverse partners under partial observability.

In response to this point, we conducted preliminary experiments in a second environment: a modified version of CoinGame (from the JaxMARL suite). In our version of the environment, the ego agent collaborates with a teammate to collect red and blue coins in a small gridworld, and is rewarded based on the total number of coins collected (i.e. fully cooperative reward function). While the ego agent can collect both coin colours, the teammate is only able to collect either red or blue – and, analogously to our Overcooked setup, the ego agent has an additional action allowing them to control which colour coins the teammate attempts to collect. Consistent with our results on Overcooked, we find the following:

• RNN agent with influence + trained alongside both teammate types outperforms RNN without influence, RNN trained only alongside a single partner, and MLP (in terms of total coins collected) (Table 1)
• In evaluation episodes with the influence RNN agent, the teammate spends close to 80% of their time pursuing the “correct” coin colour (relative to a ~50% baseline) (Table 1)
• Teammate types are more (linearly) recoverable from the hidden states of RNN agents with influence than those without (Table 2)

Table 1 (mean/std taken over 5 training seeds and 1000 eval eps):

Table 2 (mean/std taken over 5 training seeds and 1000 eval eps):

These results suggest that the phenomenon is not limited to Overcooked, but instead reflects more general properties of cooperative RL under the right environmental pressures. We will expand upon these results in the revised manuscript.

2. Handling more complex or overlapping subtasks

Our approach can scale to more complex or overlapping subtasks, but this depends on two key factors: (i) whether the environment applies sufficient pressure to represent nuanced or entangled partner traits (rather than allowing simpler heuristics), and (ii) how observable those traits are in the teammate’s behaviour. If partner traits are behaviourally identifiable and reward-relevant, we expect implicit modelling to scale accordingly. We have clarified this in the revised manuscript.

3. Related work

We appreciate this point and will expand the related work section to better situate our contribution within partner modelling, meta-learning, representation learning, and cognitive science. To support the view that prediction is central to intelligence, we are incorporating several key references, including:

• Clark (2013) – Whatever next? Predictive brains, situated agents, and the future of cognitive science.
  Behavioral and Brain Sciences, 36(3), 181–204.
  Argues that cognition and action are driven by hierarchical prediction.

• Byrne & Whiten (1988) – Machiavellian Intelligence: Social Expertise and the Evolution of Intellect in Monkeys, Apes, and Humans.
  Oxford University Press.
  Suggests human intelligence evolved for social prediction, motivating partner modelling.

• Harsanyi (1967) – Games with Incomplete Information Played by “Bayesian” Players.
  Management Science.
  Formalises agents as reasoning over others’ hidden traits.

• Wang et al. (2016) – Learning to Reinforcement Learn. Proceedings of the 38th Annual Conference of the Cognitive Science Society (CogSci), 1315–1320.
  Demonstrates that recurrent policies can learn internal learning algorithms, enabling fast adaptation to novel tasks—supporting our use of recurrence for implicit partner modelling.

• Grosz & Kraus (1996) – Collaborative Plans for Complex Group Action.
  Highlights that effective collaboration requires commitment to shared goals, even under decentralised or partial knowledge.

We are happy to incorporate any additional suggestions for relevant work.

4. Connection to LLMs and RLHF

We thank the reviewer for raising this connection and have added a discussion to the manuscript. Our findings suggest a parallel between emergent partner modelling in recurrent agents and the internal representations learned by large language models (LLMs) trained via reinforcement learning from human feedback (RLHF). In both cases, models receive only a scalar reward signal (team reward in our case, human preference in RLHF) and nonetheless develop internal representations that support effective interaction. Notably, this modelling arises without explicit objectives for partner inference or prediction, indicating that social reasoning can emerge as a byproduct of reward optimisation under the right pressures.

A particularly relevant point of contact lies in the role of influence. We find that partner modelling emerges more robustly when agents can influence partner behaviour—for example, by assigning tasks. This mirrors findings in LLMs, where richer user modelling appears in interactive, multi-turn settings, where the model’s outputs shape future feedback (Ouyang et al., 2022;). Together, these results point to a general principle: when a learner’s reward depends on the actions of others and the learner can influence those actions, internal models of others are likely to emerge—even in the absence of explicit modelling objectives or architectural biases.

5. Methodological novelty

We appreciate this observation — we agree that both the analysis methods and agent architectures are standard, and we chose them intentionally. Our goal was to demonstrate that structured, partner-specific representations can emerge robustly — not as an artefact of architectural choices or analysis techniques, but as a consequence of the collaborative training setting. That said, we agree that exploring richer architectures or novel analysis methods would be a valuable direction for future work. We will add a note on this in the revision.

6. Explicit vs implicit partner modelling

We did not compare against explicit architectures, as our aim was to test if implicit modelling emerges in standard agents without scaffolding. We would expect explicit approaches to perform better on the specific traits they target, but the strength of the implicit RNN approach lies in its flexibility: it can represent any partner attribute that supports reward maximisation. A direct comparison — in terms of adaptation speed, value estimates, and generalisation — would be a valuable direction for future work. We will add a note on this in the revision.

7. Transformer architecture

Thank you for this point—it's a good suggestion and we agree it opens up a very interesting direction for future work. Transformers could support additional analyses, such as examining attention over grid cells to investigate whether the agent implicitly tracks the other’s position or trajectory.

In this work, we chose an RNN because it is a standard architecture for memory-based agents in MARL (Foerster et al., 2016), and offers simple, interpretable internal dynamics. Our aim was to explore whether structured representations emerge without strong architectural inductive biases, and the RNN is well suited to that goal. While transformers could also be applied here, they typically require larger datasets or pretraining to be effective (Chen et al., 2021), which makes it less suitable for our setting. Additionally, to our knowledge, transformers are mainly used in offline RL (Chen et al., 2021). That said, using transformers to probe agent representations (particularly via attention) would be a promising direction for future work, and we note this in the revised paper.

8. Formatting

Thank you for this point - we have updated the formatting of the paper accordingly to make the figures easier to read and to help include more results.

References

Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., & Abbeel, P. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. Advances in Neural Information Processing Systems, 34, 15084–15097.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Christiano, P. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.

Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016). Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Advances in Neural Information Processing Systems, 29.

Thank you for your thoughtful and detailed engagement with the paper. We appreciate your recognition of the paper’s originality, clarity, technical quality, and broader significance. We're pleased the experimental design and partner modelling results were well received. Below, we respond point by point and outline revisions we have made or are making to clarify and expand on the points you raised, which we believe further strengthen the paper.

Questions

Q1. Please elaborate on the agent architecture design

Thank you for the question - we have updated the appendix to clarify this. The architecture includes both input and output FC layers: after the CNN processes the observation, an FC layer maps the embedding to the GRU’s hidden size. The GRU output is passed to separate two-layer MLPs for the actor and critic, producing action logits and a scalar value, respectively. Observations are pre-processed to give each agent a local, self-centred view, and the CNN output is normalised before it enters the GRU.

Q2. Figure 2A: Why does the MLP perform better in the five-by-five environment?

This is a good point, and we agree that it indicates that partner modelling is environment dependent. We have updated the manuscript to include an acknowledgement and brief discussion of this idea.

In the five-by-five layout, spatial coordination is less critical than in cramped_room variants (due to the simpler and more open geography), making feedforward models more competitive. Additionally, these plotted results are from the fully observable setting, where agents can often infer their partner’s behaviour from the current state alone — reducing the benefit of temporal modelling. In contrast, the performance gap is larger in the partial and blind settings, where temporal reasoning becomes more important. Another possible factor is that agents are evaluated with novel partners. RNNs may overfit to training partners, whereas MLPs, lacking memory, may generalise better when spatial context is sufficient.

Q3. Figure 3A: Was there any further investigation of the no-influence RNN in cramped_room_v4?

Yes we did investigate this but we were not able to find any explanation for the apparent structure of the no-influence UMAP plot for cramped_room_v4. We suspect that it is likely an artifact of the UMAP procedure (a phenomenon which is not uncommon) – as evidence for this, we created corresponding plots using PCA and t-SNE in place of UMAP, and did not see a similar grouping appear (while the structure of points in the influence case was preserved to at least some degree).

Q4-5. Linear probing and throughput after partner switch

Thank you for this helpful point — we are revising the manuscript to clarify why the throughput plateau is lower post-switch. The observed drop in throughput after 300 epochs is actually due to an asymmetry in the speed dynamics between the ego agent and the partner agent when their roles are reversed. In the runs plotted, the partner performs serving soup quickly (1 action per step) and preparing ingredients slowly (1 action every 7 steps). Later, this pattern reverses. The ego agent acts at a constant rate (1 action every 3 steps) for both tasks.

Serving soup involves a longer sequence of actions (e.g. retrieving a bowl, ladling soup, serving) than placing onions in a pot. The system performs better when the faster agent handles the more complex task offsetting its length and improving soup throughput.

The rise in the curve reflects the ego agent adapting to this dynamic—learning to coordinate with a changing partner. Once the throughput plateaus, it indicates the agent has successfully adapted to the partner’s new behaviour.

Q6. Appendix A.3.2 — Was the hypothesis about task allocation tested?

Yes, Experiment 3 (the blind agent setting) serves as a partial test of this hypothesis. In this setting, the ego agent was paired with partners of varying competence, and the blind agent had to infer its partner’s ability and decide whether to complete the entire task alone or specialise in a subtask.

Limitations

Two agents only

This is an interesting direction for future work, and we have updated the manuscript to discuss it more explicitly. There is no reason to believe that the same kind of implicit representation would not scale to more than one teammate, provided the environment imposes sufficient pressure—i.e. all teammates’ behaviour is relevant to task completion. That said, inference becomes more demanding with additional agents, as more evidence is required to accurately identify their individual properties. The ego agent may rely on useful shortcuts, such as modelling the average behaviour of its teammates or inferring which teammate is best suited to each subtask.

Two learner agents

Great question — we expect that two learner agents could converge to similar strategies through mutual adaptation, though likely via different dynamics. Using a fixed-policy partner lets us focus on the specific question of how the ego agent models and adapts to its teammate. This is also related to the next question about the level of ego agent control/influence over their teammate; if both agents are learning, then it wouldn’t make sense to give one agent control over the other’s subtask, and so we would require a more nuanced mechanism for determining subtask allocation. We have added a brief discussion of this point in the revised manuscript as a direction for future work.

Direct control

This is a fair point. We used an extreme form of ‘influence’ to give us the clearest/simplest test of whether partner modelling emerges driven by environmental pressure (i.e. there is a very strong link between ability to represent teammate properties and overall reward). A more realistic setting might involve some form of communication, where the partner learns to follow (or ignore) high-level instructions (and this could even be bidirectional). We have updated the discussion to highlight this as a direction for future work.

Other Comments

Thank you for these helpful suggestions. We have revised Figure 1 to clarify the meaning of the speedometers and avoid potential misinterpretation. We have also cited JAX at first mention, corrected the typo on line 221, and standardised usage of “w.r.t.” throughout. Moreover, we have clarified the linear probe architecture in the text: it consists of a single-layer linear decoder mapping the GRU hidden state to a scalar output (speed prediction).

Regarding Figure 3B, thank you for flagging this. We're unsure which specific gap in the confidence intervals is being referred to — we’d be grateful for clarification so we can investigate further.