Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques

W1. The experiments are conducted on a limited range of tasks.

Thank you for the advice! In the updated paper, experiments on Overcooked are added. For other algorithms, experiments with MAIQL and MABCQ are included. These two algorithms are commonly used in offline MARL papers. See results below and more details in Section 6 of the updated paper.

Q3 & Q4. Inconsistency between Table 2 and claims.

We have corrected the corresponding statement in the updated version of our paper.
The current data is accurate, but the claim "in more challenging environments..." was based on data from a previous version. We updated the experimental data before submission but mistakenly submitted an outdated text version.

According to the earlier data, the expert in the tag environment was relatively weaker, and a more diverse dataset indeed resulted in higher returns. However, for fairness, we switched to a stronger expert, which significantly improved the results of imitation learning (high-$\beta$ settings) while reducing the impact of diversity on the improvement of the reward model.

Q5. Table 3: Why does setting to a magnitude as large as 100 yield such good results?

Thank you for pointing this out. The original paper missed the explanation. In our actual experiments, we clipped $\beta \log \pi(s,a)$ to $[-10, 1]$, which was not mentioned. The clipping serves to smooth out the differences in penalty terms among behaviors with relatively high density in the dataset. After clipping, excessively high $\beta$ values no longer dominate the loss, and the performance for higher $\beta$ becomes very similar. We have added this clarification in the updated version of the paper.

Q6. Figure 2: What does the x-axis represent?

The X-axis in Figure 2 represents time, so all the graphs show the reward/predicted reward curves over time. Their integral corresponds to the return.

Typos in the paper

Thank you for pointing out the typos in our paper, we have corrected them now:

• Figure 1: Replace $\pi_{ref}$ with $\pi_b$.
• Line 300: Delete the blue words.

W1. Reward regularization idea in previous works

Our work introduces reward regularization specifically within the context of preference-based MARL, addressing challenges due to the complexity of multi-agent interactions and long-horizon dependencies. While different from prior works that use adversarial adjustments [5] or Bellman-consistent pessimism [6] in general RL settings, we use MSE loss for reward regularization. However, we share a similar high-level idea to stabilize reward model training. We have incorporated these insights into the related work section.

W2. Could the authors include more baseline methods and more tasks for comparison?

As an initial work in preference-based MARL, to the best of our knowledge, there are no directly comparable traditional approaches.
The most straightforward comparison is to combine a reward model with offline RL algorithms. In the updated paper, experiments on Overcooked are added. For other algorithms, experiments with MAIQL and MABCQ are also included. These two algorithms are commonly used in offline MARL papers. See results below and more details in Section 6 of the updated paper.

W3. Could the authors extend the analysis to accommodate general function approximations, or clarify how the experimental setup meets the requirements of linear function approximation?

We agree that our theory based on linear assumptions does not perfectly match our experiments. However, we want to emphasize that our theory provides a fundamental understanding of offline preference-based MARL and guided the experiment design, where we compared datasets with different levels of diversity. Extending our analysis to general function approximation would be an interesting future direction.

W4. In Line 276, the reference to "an approximate Nash equilibrium policy" in the theorem lacks clarity

We provided the complete theorem in the appendix (Theorem 4 in Line 1074). The approximation error is bounded by the inverse of the covariance norm, which will have an $O(\sqrt{n})$ rate if we have $n$ samples from a fixed distribution. This type of bound is also widely adopted in offline RL literature [1, 2].

W5 & Q2. The authors should expand on the implications of the derived bound and compare their results with existing theoretical findings in the offline RL and MARL literature.

For preference-based MARL, we extend the analysis to accommodate preference-based datasets, which differ from standard offline datasets with fixed state-action-reward tuples. This leads to different covariance matrices and uncertainty measures compared with standard offline MARL. Additionally, we establish that unilateral coverage is sufficient for learning approximate Nash equilibria, deriving bounds that explicitly account for preference-based dynamics.

Q1. This paper analyzes the RLHF setting; however, the definition of the performance metric remains unchanged from the RL setting without KL regularization. Could the authors provide further clarification on this?

In our study, we adopt the standard performance metric from traditional RL without incorporating KL regularization. We aim to propose a basic setting for preference-based MARL, aligning with the general cases. As KL regularization is an efficient practical method, it is also natural in RL literature to use a performance metric without additional regularization terms [3, 4]. We acknowledge that incorporating KL regularization can be beneficial in certain contexts, and we plan to explore its integration in future work.

Typos and writing clarity

Thank you for pointing out the typos and writing clarity problems. We have removed the blue words.
We have enriched our related work section with the recent advancements in MARL and RLHF as you mentioned.

[1] Jin, Ying, Zhuoran Yang, and Zhaoran Wang. "Is pessimism provably efficient for offline rl?." International Conference on Machine Learning. PMLR, 2021.
[2] Zhong, Han, et al. "Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets." International Conference on Machine Learning. PMLR, 2022.
[3] Longyang Huang, Botao Dong, Ning Pang, et al. Offline Reinforcement Learning without Regularization and Pessimism. TechRxiv. June 07, 2024.
[4] Le Lan, Charline, Marc G. Bellemare, and Pablo Samuel Castro. "Metrics and continuity in reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021.
[5] Mete, Akshay, et al. "Reward biased maximum likelihood estimation for reinforcement learning." Learning for Dynamics and Control. PMLR, 2021.
[6] Xie, Tengyang, et al. "Bellman-consistent pessimism for offline reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 6683-6694.

Thank you for your review. Please find our responses to your comments below.

WP1. Q1. & Q2. There is no actual human feedback involved.

You are correct that our current experiments rely on simulated preferences rather than true human feedback. To better reflect this, we will change the title to "Preference-Based Multi-Agent Reinforcement Learning" to clarify our focus.

WP2. The theoretical results rely heavily on assumptions that may not hold in practice.

We agree that our theory based on linear assumptions does not perfectly match our experiments. However, we want to emphasize that our theory provides a fundamental understanding of offline preference-based MARL and guided the experiment design, where we compared datasets with different levels of diversity. Extending our analysis to general function approximation would be an interesting future direction.

Q3. The paper proves that their theoretical algorithm converges to Nash equilibria, but the practical implementation uses different algorithms (VDN-based) with no theoretical guarantees.

Our theory demonstrates the data coverage conditions fundamentally needed even in the basic linear function approximation setting. For more real-world problems requiring general function approximation, such as neural networks, we chose VDN to solve these problems. Our experiments showed that unilateral coverage and more diversified datasets improve performance, verifying our theoretical insights.

WP4. & Q4. With only 5 seeds, the reliability of their comparisons is questionable.

Thank you for raising this point. We have rerun our experiments with 10 seeds and updated the data in the revised version. We observed that the variance in performance across different seeds remains very low, which we believe provides sufficient robustness for our findings.

WP4. & Q5. The differences appear small relative to their standard deviations.

Some of the comparative results indeed lack persuasiveness, such as the examples you mentioned (-20.98 ± 0.56) and (-21.11 ± 1.16). As a result, our practice of bolding data with only minor advantages in the table may have been misleading. In the revised version, only comparison results with a p-value less than 0.05 in the significance test are bolded. Additionally, more precise language is used to describe the empirical conclusions.

Q6. How well does your approach scale with increasing numbers of agents?

We've conducted an experiment to test the scaling problem in Appendix B3 (B4 in the updated version). A brief conclusion is that, while our current approach manages the scaling of agents without introducing new problems, it does not specifically address the inherent issues of instability and complexity that are well-documented in traditional MARL. We added a paragraph discussing this in the main text of the updated paper.

Q7. In cases where mixed-skill policies outperform pure expert policies, can you verify that this reflects genuine improvement rather than issues with reward modeling?

In preference-based RL, reward modeling is an integral part of the complete algorithm, and our theoretical analysis does not separate reward modeling from the subsequent RL Oracle. For example, in Theorem 4 (line 1074), the approximation error is bounded by the inverse of the covariance norm, which depends on the diversity of the dataset. Therefore, the fact that a mixed dataset can provide a better reward model than a pure dataset is itself a genuine improvement.

Q8. & WP3. Have you tested MARL algorithms other than VDN? & The experimental evaluation, while systematic, is limited to relatively simple environments in the Multi-Agent Particle Environment (MPE) framework.

Thank you for the advice! In the updated paper, experiments on Overcooked are added. For other algorithms, experiments with MAIQL and MABCQ are added. These two algorithms are commonly used in offline MARL papers. See results below and more details in Section 6 of the updated paper.

Thank you for your review. Please find our responses to all your comments below.

W1. W2. & Q1. Testing on more realistic and complex environments and conducting ablation studies with other MARL algorithms

Thank you for the advice! In the updated paper, we added the Overcooked environment to address the lack of long-duration, strategy-intensive task environments. Furthermore, we additionally tested two more algorithms: Multi-agent IQL (MAIQL) [1] and Multi-agent BCQ (MABCQ) [2].

• Remark 1: The results on Overcooked highlight the importance of dataset diversity and unilateral data.

• Remark 2: Since preference-based MARL is already a new and complex setup, we intentionally selected simple implementations for both the environment and the algorithms to avoid introducing additional uncertainties. We did not test on-policy algorithms like PPO because they require the replay buffer to be collected by the currently optimized policy, making them unsuitable for direct application in offline reinforcement learning, which is the focus of our setting. On the other hand, the QMIX algorithm is more suited for environments involving a large number of agents and complex policies. Therefore, for additional experiments with more algorithms, we adopted Multi-agent IQL (MAIQL) [1] and Multi-agent BCQ (MABCQ) [2], which are simpler to implement but commonly used in offline MARL papers under the centralized-training decentralized-execution (CTDE) framework.

Q2. What challenges do you anticipate on more complex MARL benchmarks?

One of the primary challenges is reward modeling. In more complex environments (e.g., SMAC), especially in longer-horizon, continuous settings, the lack of supervisory signals becomes more apparent. Since reward models learned through preference learning often exhibit significant errors, agents tend to perform poorly in low-robustness tasks that emphasize fine-grained operations. This contrast is evident in the experimental results of MPE and the newly added Overcooked environment: in Overcooked, the agents can generally achieve performance close to the expert level, whereas in the three MPE environments, there is a noticeable gap between the two.

Q2. How sensitive might performance be to the choice of hyperparameters $\alpha$ and $\beta$?

An ablation study on $\beta$ and $\alpha$ can be found in Section 6 and Appendix B.5. The choice of $\beta$ is highly robust. Due to clipping, excessively large $\beta$ values will not dominate the entire reward function. As a result, larger $\beta$ values almost never degrade the agent's performance in our experiments. This allows us to increase $\beta$ with relative confidence. Therefore, we generally recommend setting $\beta$ to a value between 10 and 100.

Q3. What policy was used to generate responses for collecting preference feedback?

We used VDN for the three MPE environments and IPPO on Overcooked to train the expert policy. We tested VDN, IPPO, MAPPO, and QMIX on all the environments and took the best agent trained as the expert. Suboptimal trajectories were collected using checkpoints saved during the training of the expert policy. More details can be found in Section 6.

Q4. How was the preference feedback collected?

We simulated the preferences with the Bradley-Terry Model. Details can be found in Section 6.