Learning Preferences without Interaction for Cooperative AI: A Hybrid Offline-Online Approach

Dear reviewer 5sd7:

We sincerely thank you for the time and effort dedicated to reviewing our work. Your comments are highly appreciated and help us further improve the quality and clarity of our paper.

We address each of your points in detail below:

Response to W1 (“work on generating diverse policies/conventions”): We thank the reviewer for raising this important point. We will include a discussion of relevant methods on generating diverse policies and conventions in the related work section of the camera-ready version. Regarding the recent body of work on generating diversity through cross-play minimization, we selected two representative and validated methods from ZSC-eval [1]: TrajeDi [2] and MEP [3]. These methods have been shown to be effective in the Overcooked environment. For evaluation, we directly used the released checkpoints from ZSC-eval. Notably, these checkpoints were trained with a policy pool size of 36 (agents with standard preferences), which is larger than ours (18) and thus has stronger generalization potential. The experimental results on layout random1 are as follows:

These comparisons further support the necessity of explicit preference learning, as our method consistently performs well even when evaluated against strong ZSC baselines.

Further Response to W1 (on KL-constrained PPO and PPObc): In our setting, the number of available offline trajectories is extremely limited (around a dozen), making it difficult to estimate KL-constrained rewards reliably. Many states encountered during training are out-of-distribution (i.e., not present in the offline data), especially at the early stage. This leads to large estimation errors in KL-constrained methods due to the mismatch in dynamics between the offline trajectories and the policy pool policies. Our method avoids this issue by decoupling preference learning and generalization: the online recovery stage in EAR focuses solely on regaining generalization ability, without relying on preference alignment.

Regarding PPObc, the core idea is to first fit a human proxy, and then train the cooperative agent via reinforcement learning to coordinate with this proxy. While this may work in simple or two-player settings, modeling the proxy accurately becomes significantly more challenging in multi-agent or more complex environments. In contrast, EAR does not rely on accurate modeling of the human proxy, and is able to simultaneously generalize and align with human preferences even when the dynamics in offline data and online recovery differ.

Response to W2 (“This work only conducts evaluations against human proxies”): We acknowledge this limitation. Due to budget and time constraints, we have not yet conducted experiments involving real human participants. We appreciate the reviewer’s feedback, and we consider this an important direction for future work.

Response to Notation Error (1): Thank you for pointing out the distinction between AHT and ZSC. We will include a discussion of their relationship in the camera-ready version. Our current use of ZSC terminology follows the convention adopted in prior works such as FCP, MEP, HSP, and ZSC-eval. That said, we agree that the definition may require further clarification, and we will revisit it carefully in our revision to ensure conceptual precision.

Response to Notation Errors (2), (3), and (5): We appreciate the reviewer for identifying these issues. These are either minor typos or instances of unclear writing, and we will conduct a thorough review of the entire paper to eliminate such avoidable errors and improve the overall clarity.

Response to Notation Error (4): Thank you for pointing out the imprecise use of the term “pretraining” in our paper. In our setting, pretraining refers to the process where the cooperative agent is trained via RL with policies from the policy pool to improve its generalization ability, i.e., its capacity to collaborate with a variety of agents to complete tasks in the environment. We acknowledge that this use of pretraining may differ from its usage in NLP, where pretraining typically involves learning from large-scale corpora to capture syntax and semantic knowledge. However, we view the intent as analogous—just as NLP models gain basic language competence through pretraining, our cooperative agent is meant to acquire foundational understanding of task structure and partner dynamics in Overcooked via this pretraining process. We will make this distinction clearer in the camera-ready version to avoid potential confusion.

Response to Q1: Thank you for the question. We clarify that we did not use any poor-quality or noisy data in our experiments. However, we acknowledge that our current method does not leverage unlabelled or low-quality preference data, which we regard as a limitation and explicitly discuss in Appendix E. That said, a key contribution of our work is that EAR achieves a strong balance between generalization and preference alignment using only a small number of positive-labeled trajectories. We view this label-efficient approach as a meaningful step toward more practical preference learning in multi-agent settings.

Response to Q2 and Q3: In our current setup, we do not attempt to model a preference-based reward function. The primary reason is the extremely limited amount of offline preference data—only around 10-20 trajectories. Under such sparse data, reward learning methods (e.g., inverse reinforcement learning) are prone to extrapolation errors due to a large number of out-of-distribution states, which leads to unreliable or unstable reward estimation.

Similarly, using KL-constrained RL to learn from these offline trajectories faces challenges: the dynamics of the offline trajectories differ significantly from those induced by the policy pool, resulting in large initial KL divergence and unreliable gradients, especially early in training. This can cause performance degradation due to compounding errors. In contrast, our method avoids this issue by separating preference alignment and generalization: EAR’s online recovery phase is used solely to restore generalization ability, while preference information is distilled only from the limited set of labeled trajectories.

Response to Q4: In the main paper, “checkpoint” refers to the best-performing checkpoint of FCP. The term sub-optimal is used in two senses: (1) Under the FCP setup, the biased human agent used for supervision is no longer available, and its behavior closely resembles the offline preference trajectories. This suggests that there is limited room for improvement beyond the offline data. (2) More broadly, in the absence of the human supervisor, other methods from the ZSC literature such as TrajeDi or MEP may achieve better coordination than FCP when evaluated under standard ZSC metrics. However, when evaluated under our preference alignment objective, these methods do not significantly outperform FCP, suggesting that coordination alone is insufficient for preference learning.

If you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.

[1] ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination (NeurIPS 2024)

[2] Trajectory Diversity for Zero-Shot Coordination. (PMLR 2021)

[3] Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination (NeurIPS Cooperative AI Workshop 2021)

Thank you for the clarifications and detailed rebuttal. I still feel that other comparisons would strengthen the paper, but the TrajeDi and MEP results are promising (though to clarify these are not "cross-play minimization" techniques, but instead achieve diversity in different ways).

I think the paper would've been stronger if it compared with the existing RLHF work (KL-constrained PPO and PPO-bc), especially comparing the performance versus number of human samples, but I think this work is acceptable as it stands so I will raise my score.

Dear reviewer 7F41:

We sincerely thank you for the time and effort dedicated to reviewing our work. Your comments are highly appreciated and help us further improve the quality and clarity of our paper.

We address each of your points in detail below:

Response to W1 (“human may still have corrections or different preferences”): We agree with the reviewer’s observation. Indeed, the fact that “a trajectory is endorsed by the human” does not imply that the human will always make identical decisions in the future. We have taken this uncertainty into account when designing our benchmark. Specifically:

1. To encourage stochasticity and avoid overfitting to specific behaviors, we generate and evaluate offline trajectories by sampling from parameterized agents, rather than using deterministic policies.

2. As stated in Lines 267–269 of the paper: “we retain the trajectories whose biased return reaches at least 80% of the maximum return under that preference, and treat them as liked trajectories. The corresponding human-side policies in these trajectories serve as test-time collaborators for evaluating the cooperative agent.” Each human preference is associated with a well-defined reward function, consistent with the assumption made in ZSC-eval [1]. Multiple human proxy agents may be used to evaluate under the same preference, further capturing the diversity and variability of human decision-making in real-world settings.

Response to W2 (Concern about computational cost): We appreciate the reviewer’s concern regarding computational efficiency. In our experiments, we set the rollout size to 200 and the trajectory length to 400, with a total environment step budget of 20 million. This implies that EAR performs at most 250 rollouts throughout training.

All experiments were conducted on a single NVIDIA RTX 2080 Ti GPU. Under the random1 layout, style A, the average training time per step is as follows:

As shown in the table above, EAR requires approximately 2 to 4 hours of execution time per style on a single RTX 2080 Ti GPU.

Regarding the question “how would these change for a more diverse policy pool”, we sincerely thank the reviewer for raising this important point. Exploring the scalability of our method under a more diverse policy pool is a valuable direction, and we consider it an important aspect for future work.

Response to W3 (Unclear description in Section 5.3): We apologize for the unclear description in Section 5.3. Our intent was to first present the overall performance of our method, and then to provide a deeper understanding of the learned preferences by visualizing event-count vectors (one per trajectory) using UMAP. The resulting clusters reflect the impact of preference learning on the cooperative agent’s behavior. We appreciate the reviewer’s feedback, and we will improve this section by including more fine-grained, preference-specific analyses in the camera-ready version.

Response to Q1: The pretrained policies available to the agent—i.e., the policies in the policy pool—are not subject to any explicit conditioning or constraints. In our experimental setup, these policies are trained via self-play under the standard environment reward function, meaning their behavior is solely driven by reward maximization. This design choice ensures that no information about human preferences is leaked into the policy pool, maintaining a clear separation between environment reward and human preference. Regarding “how much should policies be reflective of / adhere to the preferences in the offline data,” we determine whether a policy aligns with a given preference by computing the preference score of the trajectories it generates. This allows us to assess how well a given policy adheres to a particular reward function without requiring explicit conditioning.

Response to Q2: By lightweight behavior cloning, we refer to the fact that in our method, the behavior cloning (BC) phase requires significantly less time and computational resources compared to the online recovery stage.

If you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.

Response to Paper Formatting Concerns: Thank you for pointing out the formatting concerns. We will carefully review all figures and thoroughly check the entire paper to ensure that all formatting issues are resolved in the camera-ready version.

If you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.

[1] ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination (NeurIPS 2024)

Dear reviewer 7shS:

We sincerely thank you for the time and effort dedicated to reviewing our work. Your comments are highly appreciated and help us further improve the quality and clarity of our paper.

We address each of your points in detail below:

W1: HSP as a Baseline

Our work aims to simulate a realistic setting where we are given an available policy pool and environment, along with only a very limited number of biased human trajectories (possibly as few as a dozen). The key goal of our method, EAR, is to effectively infer the preferences of a biased human agent—even when this agent is not part of the policy pool—using only a small amount of offline data. This setup closely mirrors practical constraints in real-world game industry scenarios. In contrast, in the Overcooked environment, if the biased human agent is also available (i.e., present in the policy pool), then the online recovery phase would encounter the same dynamics as the offline trajectories. This setting is already well addressed by existing work such as ALIGN-GAP [1]. Our work instead tackles a more challenging and realistic setting that has not been adequately addressed by prior methods.

W2: ZSC Methods:

We agree that in cases where the biased human agent is unavailable, partner diversity becomes important and has been considered in some ZSC works. However, it is important to note that these methods typically aim to maximize environment reward, not to align with human preferences—which is the central goal of our preference learning setup.

To provide a fair comparison, we reproduced TrajeDi [3] and MEP [2], two representative ZSC methods. For evaluation, we directly used publicly released checkpoints from ZSC-eval [4]. These checkpoints were trained with a policy pool of size 36 (agents with standard preferences), which is larger than ours (18), and thus exhibits stronger generalization ability. The experimental results are summarized below:

These results further demonstrate the necessity of explicitly modeling preference learning, beyond general reward maximization.

Response to Q1: In the main text, the term “sub-optimal” is used in two respects. First, in the context of FCP, the biased human agent is unavailable, and its performance in the ORACLE setting is close to that of the offline trajectories—indicating limited improvement. Second, we acknowledge that in the absence of the biased human, methods such as TrajeDi and MEP from the ZSC literature may perform better than FCP in terms of coordination. However, their performance does not show significant advantages over FCP when evaluated under the preference learning objective.

The relatively poor performance of FCP in our benchmark is indeed due to its inability to cover the space of the biased human, which is a well-known challenge in real-world game industry applications. Human preferences are inherently diverse and difficult to fully anticipate or enumerate. Regarding the “large reward perturbations”: the implicit reward induced by human preferences can deviate substantially from the environment reward. More fundamentally, since the biased human is unavailable, the dynamics encountered during online recovery differ significantly from those observed in the offline trajectories. This distribution shift makes it difficult for agents trained via FCP to adapt effectively to the biased human’s behavior.

Response to Q2: Below, we provide the average environment return of the agent trained using FCP across its policy pool, as well as its scores under the three different human preference styles:

Response to Q3: In principle, the pre-trained policy can be recurrent. However, in our experiments, we opted for a non-recurrent architecture due to implementation considerations when applying Behavior Cloning. Training with recurrent policies requires handling memory, which involves either storing hidden states for each timestep or adopting a trajectory-wise training scheme. We appreciate your thoughtful question, and we will consider extending our experiments to include recurrent architectures in future work.

If you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.

[1] ONLINE-TO-OFFLINE RL FOR AGENT ALIGNMENT. (ICLR 2025)

[2] Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination (NeurIPS Cooperative AI Workshop 2021)

[3] Trajectory Diversity for Zero-Shot Coordination. (PMLR 2021)

[4] ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination (NeurIPS 2024)

Dear reviewer YpWX,

We sincerely thank you for the time and effort spent reviewing our paper and for providing valuable suggestions for improvement. We address each of the comments point by point below.

Clarification on Section 4.2
We apologize for the confusion caused by unclear descriptions in Section 4.2 and the corresponding tables. The sentence “describe each stage in detail below” was intended to introduce the methodological description of the three stages of our approach. Specifically, the second stage involves a single round of online generalization recovery after offline behavior cloning. In Section 4.2, our goal was not only to present the detailed implementation of this second stage, but also to analyze its limitations through experiments—thus motivating the necessity of the third stage: Epoch-wise Alternation Recovery. We acknowledge that the current title of Section 4.2 may have been misleading, and we commit to revising it in the camera-ready version to better reflect the section’s content.

Clarification on Table Descriptions In Table 1, offline trajs refer to the limited number of offline expert trajectories available during training. These serve as golden trajectories and are generated by preference-aligned agent pairs that have reached a Nash equilibrium through self-play within the benchmark environment. We also apologize for the ambiguity in our phrasing of “agents that were not involved in the offline preference data” and “all benchmark human proxy agents.” Our intention was to convey that, in addition to evaluating on the agents that were involved in the offline preference data, we also conducted evaluations on agents that were not involved in the offline data, in order to assess the generalization ability of the cooperative agent. The combination of results from both sets of agents constitutes what we refer to as “all benchmark human proxy agents,” which is exactly what is reported in Table 2. We will unify and clarify the terminology in the table caption in the camera-ready version to avoid ambiguity.

On the Choice and Applicability of Baselines The methods discussed in Section 2.3 are designed for different RL settings. In our proposed setup, the task is cooperative in nature, where each partner induces a different environment dynamic. In contrast, most baselines in Section 2.3 assume that the online environment shares the same dynamics as the offline trajectories—an assumption that does not hold in our case, and which limits their direct applicability. Nevertheless, we made efforts to reproduce ALIGN-GAP [1] (published at ICLR 2025) within our setting. For reward model learning, we used the ground-truth reward function directly. Specifically, when the preference-based reward of a trajectory exceeds the threshold observed in the offline trajectories, the reward model outputs a “like” signal for that trajectory. The reproduced results under the random1 layout are as follows:

It is important to note that the ALIGN-GAP method was originally designed for single-agent settings, where the dynamics of the offline trajectories and the online environment are assumed to be the same. This key difference from our cooperative multi-agent setting, where the dynamics vary depending on the partner, makes the direct application of ALIGN-GAP ineffective. The experimental results further highlight the necessity of adopting our proposed approach.

If you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.

[1] ONLINE-TO-OFFLINE RL FOR AGENT ALIGNMENT. (ICLR 2025)