O-MAPL: Offline Multi-agent Preference Learning

We sincerely appreciate the reviewer for carefully reading our paper, offering a positive evaluation, and providing valuable and insightful comments. To address your questions, we have conducted additional experiments, which are detailed in the PDF available via this anonymous link: https://1drv.ms/b/s!AgChHLa7t5Bza5QJpMfi7YJX6PI

---

Below, we address each of your comments and questions:

Why do you switch from discounted MDPs to finite-horizon MDPs when you discuss preference-based learning in Section 4.1? Isn't it better to have a unified framework?

We sincerely thank the reviewer for this insightful comment. We initially adopted the discounted infinite-horizon MDP framework as it is widely utilized in RL and MARL literature due to its general applicability. However, for our objective of the preference-based learning, we chose to exclude the discount factor because the preference data commonly consists of partial or sub-trajectory information rather than complete trajectories. Therefore, our algorithm was developed around an undiscounted sum of rewards, which aligns closely with recent preference-based approaches in the single-agent setting. We will clearly articulate and emphasize this choice in the revised manuscript.

Related to my comment on the convergence, is it possible to add some convergence guarantees of O-MAPL? There is an alternating optimization problem being solved for both the optimal Q-function and value function. Due to convexity/concavity, I expect it to converge in finite time.

We sincerely thank the reviewer for this insightful comment. Our alternating optimization approach is based on reformulating the primary learning problem as an equivalent bilevel optimization problem, as shown below:

$max_q L(q,v,w)$ s.t. $v= \arg\min J(v)$

In this formulation, $L(q,v,w)$ is concave in $q$ and $J(v)$ and is convex in $v$. This convex-concave structure allows the alternating optimization process — where $q$ and $v$ are updated aternatively — to converge to a stationary point. This optimization method is widely adopted and well-supported in the literature on bilevel optimization.

We will clarify this point and cite relevant works on bilevel optimization to support our approach. Thank you for bringing this to our attention.

However, what is not clear to me is how the presence of WBC affects convergence.

We thank the reviewer for this insightful question. To clarify, the convergence of the Q and V functions arises from solving the bilevel optimization problem using an alternating procedure. The convergence of the WBC component, on the other hand, is theoretically supported by Theorem 4.3 under the assumption that Q and V are fixed. In practice, we train the WBC component simultaneously with Q and V updates to improve computational efficiency. Empirically, one can show that as Q and V converge, the WBC also reliably converges to the optimal policy given a sufficient number of update steps.

We will better clarify this point in the revised manuscript.

How does data coverage affect the performance of O-MAPL, that is, have you tested it in datasets where not enough "good" trajectories are present? How does it affect the performance?

We thank the reviewer for this insightful question. To address this point, we conducted additional experiments to evaluate the effect of data quality and coverage on O-MAPL’s performance. Specifically, we varied the number of preference samples, and examined cases where LLMs provided lower-quality preference labels. The additional results are reported in Tables 1, 2 and 4 in the PDF. These additional results show that O-MAPL remains robust across a range of dataset qualities, although its performance degrades gracefully as the quality of preference data decreases.

We will include these additional results in the revised manuscript to highlight this robustness.

Can you comment on the presence of the additional term in Proposition 4.5? The result indicates that the value function is not "exactly" expressed as log-sum-exp of the Q-function, but there is an additional bias. Why is that?

We thank the reviewer for this insightful question. In Proposition 4.5, we demonstrate that the local value function cannot be exactly expressed as the log-sum-exp of the local function. This highlights a key limitation of prior MARL approaches that rely solely on local functions to compute local policies or values. The fundamental reason is the interdependence among agents in cooperative settings — local policies are influenced by the joint behavior of other agents. As such, expressing them purely in terms of local and functions leads to an approximation error. We will clarify this point more explicitly in the revised manuscript.

---

We hope the above responses satisfactorily address your questions and concerns. If you have any further questions or comments, we would be happy to provide additional clarification.

We sincerely thank the reviewer for the positive evaluation and insightful feedback. Below, we provide detailed responses to your comments. To address your questions, we have conducted additional experiments, which are detailed in the PDF available via this anonymous link: https://1drv.ms/b/s!AgChHLa7t5Bza5QJpMfi7YJX6PI

---

How does the method perform when preference feedback is limited?

We thank the reviewer for the insightful comment. To address your question, we conducted additional experiments in which we systematically reduced the amount of preference data to 75%, 50%, and 25% of what was used in our main experiments. The comparison results — reported in Tables 1 and 2 in the PDF — demonstrate how our method and several important baselines perform under reduced preference feedback. As expected, the overall performance of all methods degraded with less data; however, O-MAPL consistently outperformed the other baselines across all settings, showing robustness to limited preference feedback.

Insufficient demonstration of the value decomposition module’s specific contributions.

We thank the reviewer for the comment. To clarify, we have included several variants of our method to evaluate the specific contributions of the value decomposition module. Specifically, IPL-VDN is a variant where we fix the weights in the value decomposition module and do not learn them, while IIPL represents a variant in which each local value function is learned independently per agent, without being aggregated through our mixing architecture.

In addition, we have conducted additional experiments comparing the performance of 1-layer and 2-layer mixing networks in our value decomposition approach. The comparison results are reported in Table 3 of the PDF.

Can the approach be extended to mixed cooperative-competitive scenarios?

We thank the reviewer for the question. While the value decomposition approach has potential applicability to mixed cooperative-competitive scenarios, it would require substantial adaptations to handle such complexities effectively. We consider this extension beyond the current scope but agree it is a valuable direction for future exploration.

---

We hope that the above responses satisfactorily address your questions and concerns. If you have further questions, we are happy to provide additional clarification.

We greatly appreciate the reviewer's detailed feedback and constructive questions. To address your questions, we have conducted additional experiments, which are detailed in the PDF available via this anonymous link: https://1drv.ms/b/s!AgChHLa7t5Bza5QJpMfi7YJX6PI

---

Below, we respond point-by-point to your valuable suggestions and concerns.

There are no comparisons of training curves, convergence rates, or sensitivity analyses against baselines like IIPL or IPL-VDN …

We thank the reviewer for highlighting this important point. To clarify, the curves presented in our paper (Figure 1 and additional figures in the appendix) represent evaluations during training progress. Specifically, these curves illustrate the evolution of the value function during training, comparing different approaches (O-MAPL, IIPL, IPL-VDN).

The experiments use LLM-generated preference data for training O-MAPL but do not clarify whether baseline methods (e.g., IIPL, BC) were evaluated under the same data conditions.

We thank the reviewer for the insightful comment. We would like to clarify that all baselines have been evaluated using the same preference data. We will explicitly state this point in the revised manuscript.

Additionally, there is no ablation study comparing rule-based vs. LLM-generated preferences for O-MAPL

We appreciate the reviewer's suggestion. To clarify, we already have Table 1 and several figures in the appendix which show a clear comparative analysis between rule-based and LLM-based preference data across all methods considered.

Implementation details lacked critical specifics on preference dataset generation (e.g., LLM hyperparameters) and omitted pseudocode, hindering reproducibility.

We thank the reviewer for this feedback. To clarify, we have already included several details in the appendix on our data generation methods (rule-based and LLM-based), including hyperparameters and specific prompts used to obtain LLM-generated preferences. We believe this information is sufficient for reproducibility.

Regarding the pseudocode, we agree that our algorithm description is currently quite brief. We will include a more detailed and structured presentation of the algorithm in the updated paper to improve clarity and reproducibility.

Additional experiments provided limited ablation studies (e.g., linear mixing tested only on SMAC, not MAMuJoCo) and did not address computational costs or robustness to preference noise

We thank the reviewer for raising these important points. To clarify, our experiments did apply linear mixing and our proposed value decomposition methods to both SMAC and MAMuJoCo tasks. Regarding computational costs, we have provided some details in section B.2. If the reviewer could specify which particular aspects of preference noise we have overlooked, we would be happy to provide further clarification and analysis.

The work focuses exclusively on cooperative settings, leaving mixed cooperative-competitive environments unexplored. …

We thank the reviewer for highlighting this important consideration. Our current work focuses specifically on cooperative settings. As noted in our conclusion, addressing mixed cooperative-competitive scenarios would indeed necessitate significantly different methodologies, which lie beyond the current scope. We agree that exploring these mixed scenarios is a valuable direction for future research.

The experiments do not include comparisons with non-linear mixing variants. …

We thank the reviewer for this insightful comment. We focused our experiments on the linear mixing setting. This decision was based on previous studies indicating that two-layer mixing networks perform significantly worse than one-layer linear mixing networks especially in offline settings with limited data [1]

[1] Bui et al. ComaDICE: Offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. In ICLR 2025.

To address your question, we have conducted additional comparative experiments between linear and nonlinear mixing structures. The results, shown in Table 3 in the PDF, clearly demonstrate that the linear mixing structure outperforms the two-layer structure.

The paper does not analyze the quality of LLM-generated labels.... How sensitive is O-MAPL to potential inaccuracies or biases in LLM-generated preferences?

We thank the reviewer for the insightful comment. To address this concern, we conducted additional experiments comparing the performance of our method using two versions of ChatGPT: 4o and 4o-mini (the latter being considered weaker in terms of long-term reasoning capabilities). The comparison results, reported in Table 4 in the PDF.

---

We hope that the above responses satisfactorily address your questions and concerns. If you have further questions, we are happy to provide additional clarification.

We thank the reviewer for carefully reading our paper and providing valuable feedback. Below, we address your concern in detail.

I have one concern about the correctness of the derivation of the implicit-reward method. In section 4.2, the authors proposed to design a training objective function to avoid explicit reward modeling. However, this training objective relies on the inverse soft Bellman-operator. In L(), I did not see how to guarantee the inverse soft Bellman-operator which requires the soft Q function to be a fixed point of a soft Bellman-operator. Therefore, in the training pipeline, if the updated Q function can not be guaranteed to be a fixed point of a soft Bellman-operator, then the algorithm can not provide a convergence guarantee to an optimal policy. In practice, this issue usually lead to instability or sub-optimal solutions of the implicit-reward methods [1, 2].

We thank the reviewer for the comment. To clarify, we can formally prove that the trained $Q_{\text{tot}}$ resulting from our preference-based training objective is guaranteed to be a fixed-point solution of the soft Bellman equation. The proof closely follows the structure of prior work such as IQ-Learn, and we provide a version tailored to our formulation and notation in the following.

PROOF: With the definition of $\mathcal{T}^* Q_{\text{tot}}$ as

$(\mathcal{T}^* Q_{\text{tot}})(s, a)$ = $Q_{\text{tot}}(s, a)$ - $\gamma$ $E_{s'}$ $[V_{\text{tot}}(s')]$

we can rewrite the inverse Bellman equation as follows:

$Q_{\text{tot}}(s, a)$ = $(\mathcal{T}^* Q_{\text{tot}})(s, a)$ + $\gamma$ $E_{s'}$ $\left[ V_{\text{tot}}(s') \right]$,

which implies:

$Q_{\text{tot}}(s, a)$ = $(B_r^*$ $Q_{\text{tot}})(s, a)$,

where the reward function is defined as

$r(s, a)$ = $(\mathcal{T}^* Q_{\text{tot}})(s, a)$, and $(B_r^*$ $Q_{\text{tot}})(s, a)$ is the soft Bellman backup operator defined in our paper. This result shows that $Q_{\text{tot}}$ satisfies the soft Bellman equation.

Therefore, if we train the preference-based objective using $(\mathcal{T}^* Q_{\text{tot}})(s, a)$ — i.e., via the inverse soft Bellman equation — then the resulting $Q_{\text{tot}}$ is guaranteed to be a fixed-point solution to the soft Bellman equation with reward $r(s, a)$ = $(\mathcal{T}^*$ $Q_{\text{tot}})(s, a)$.

EndProof

We will incorporate the above discussion into the revised paper to clarify the fixed-point convergence of our method.

---

We hope that the above responses satisfactorily address your questions and concerns. If you have further questions, we are happy to provide additional clarification.