FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning

Dear Reviewer SCn5,

We sincerely thank you for your close reading and constructive suggestions. We hope the responses below address your comments.

[Q1: Behavior and Role of $\mu$ During Training and Its Potential Use with Other MORL Algorithms]

In our experiments, we observe that the dual variable $\mu$, which implicitly represents the preference weights across objectives, tends to converge relatively early in the training process (e.g., stabilizing around 20k steps out of total 100k training steps). This suggests that FairDICE quickly identifies the correct trade-off priorities by the fairness objective.

However, while your suggestion of two-stage approach (using FairDICE to find an optimal $\mu^{\\ast}$ and then plugging it into another high-performing linear MORL algorithm) sounds appealing, this direction is non-trivial due to the tight coupling between the policy optimization and the preference weight optimization in our framework.

The key point is that the equivalence we established in Section 5.2 holds specifically for the regularized setting. The optimal weights $\mu^{\\ast}$ are optimal only for the regularized problem. Without regularization, as we show in Appendix B, the optimal policy for Fair MORL (which can be stochastic) can be fundamentally different from the one found by linear MORL (often deterministic), even if they use the same weights. Therefore, one cannot simply take the $\mu^{\\ast}$ from our regularized FairDICE and plug it into a different family of offline RL algorithms (like CQL or IQL) and expect it to remain optimal. Those algorithms employ entirely different regularization principles to handle distribution shift. The optimality of $\mu^{\\ast}$ is tied to the specific DICE-style optimization landscape.

[W2, W3: Notation Clarity for Bellman Flow Constraint and Minor Typographical Corrections]

We sincerely appreciate your suggestion regarding the use of “(2)” to refer to the Bellman flow constraint. To avoid potential confusion with subscripts or equation components—especially in the definitions of P2, P3, and P2-reg—we will adopt a clearer notation. Specifically, we will use $BF(d) = 0$ to denote the Bellman flow constraint after it is formally defined in the paper.

We will also fix the following typos in the updated manuscript:

• Line 158: Fixing the summation expression by rewriting as $\sum_{s}\nu(s)\sum_{\bar{s},\bar{a}}T(s|\bar{s},\bar{a})d(\bar{s},\bar{a})$.

• Line 159: Revising $\\sum_{i}\\mu_{i}k_{i}+u_{i}(k_{i})$ to $\\sum_{i}(\\mu_{i}k_{i}+u_{i}(k_{i}))$ to ensure correctness.

Dear Reviewer s6Th,

Thank you for taking the time to review our work. We greatly value your feedback and aim to address your concerns thoroughly in the following response.

[W1,Q1:Limitations of Alpha-Fairness with Negative Returns and Recommendation to Consider Alternative Welfare Functions like Gini or WOWA]

We agree with the reviewer’s observation that the welfare functions used in our paper can become infeasible when negative returns are present, particularly functions like $\log(x)$, which are undefined for non-positive values. A practical workaround for handling negative returns is to define the utility function piecewise. For the non-positive domain, we can use a concave function that handles negative inputs, and attach it to the original $\log(x)$ utility function. For example, we can define the utility for $x<1$ as a quadratic segment, say $-0.5(x−2)^2 + 0.5$, and attach it smoothly to $\log(x)$ for $x \geq 1$. By doing so, we can ensure the function remains continuous, differentiable, and concave while being well-defined for all real-valued returns. Although this modifies the original Nash Social Welfare (NSW) objective slightly, it preserves its core fairness-promoting properties and can provide meaningful and robust solutions in practice.

Alternatively, one can consider general-purpose welfare functions such as the Generalized Gini Function (GGF) and the Weighted Ordered Weighted Averaging (WOWA), which naturally handle negative returns. However, there are two structural differences that make optimizing rank-based objectives such as GGF or WOWA directly within the FairDICE framework nontrivial.

First, the primary challenge with rank-based objectives like GGF or WOWA is the instability and optimization difficulties this introduces. These methods define the welfare function based on the sorted order of the objective returns. Since any off-policy estimation (OPE) of returns from a fixed dataset is inherently noisy, even small estimation errors can cause the predicted ranks of two closely-valued objectives to flip. Furthermore, this rank-switching can make the overall objective function non-smooth or even discontinuous with respect to the policy being optimized. This can lead to unstable learning signals, making it challenging to optimize particularly in the offline setting. It is likely for this very reason that the works for GGF and WOWA mentioned by the reviewer, do so exclusively in the online setting.

Furthermore, this practical challenge points to a difference in how these frameworks encode fairness. GGF and WOWA rely on explicit, discrete sorting to determine an objective's importance. In contrast, FairDICE uses a continuous and implicit mechanism: preference is not determined by rank, but is encoded directly into the smooth curvature of the concave utility function. This yields naturally smooth and stable prioritization based on an objective's current return level, which can lead to more stable optimization.

We appreciate the reviewer’s suggestion and agree that exploring alternative welfare formulations, particularly those that accommodate negative returns, is a promising direction for extending FairDICE to broader real-world scenarios. Such formulations could enable more stable and robust optimization of welfare in environments with high variance or unbounded return scales.

[W2, Q2: Justification for Using DICE-RL Over Other Offline RL Methods Like CQL or IQL, and Potential for Integration]

Our choice of DICE-RL framework was driven by the unique structural requirements of Fair MORL objective, which are not easily addressed by value-based methods like CQL or IQL.

The key advantage of DICE-RL is its formulation as a convex optimization problem over the space of stationary distributions (d). This structure is well-suited for our objective, $\sum_{i} u_i( J_i )$ where the expected returns $J_i$ are linear functions of the distribution $d$: $J_i = E_d [r_i]$. This allows us to use convex optimization techniques (i.e., introducing slack variables) to handle the concave utility function $u_i$ in a principled way, circumventing the difficult nested expectation problem.

In contrast, adapting value-based methods to non-linear fair objectives is fundamentally challenging. A naive approach might be to learn separate Q-functions $Q_i$ for each objective, and then derive a greedy policy. However, unlike standard linear scalarization, the standard policy improvement guarantee breaks down for non-linear objectives due to the fact that $\sum_{i} u_i( J_i( \pi ) )] \neq E_{\pi}[ \sum_{i} u_i( J_i(s,a) )]$. Because these two quantities are not equivalent, there is no guarantee that optimizing the latter will improve the former (our true goal). Therefore, naively applying value-based methods would fail to learn truly optimal fair policy. It remains unclear how to optimize a fairness-driven policy when the scalarization is non-linear.

In light of such a challenge, investigating how to adapt SOTA offline RL algorithms like CQL and IQL to the fairness-aware MORL setting is a highly promising direction for future work. It has strong potential to improve the practical performance and scalability of fair MORL methods in complex environments.

[W3: Concern About Sensitivity to Hyperparameter $\beta$ and Need for Practical Guidance on Its Selection]

We agree that the hyperparameter $\beta$ plays a critical role in balancing the trade-off between welfare maximization and distributional shift, and as such, it can influence the final performance of FairDICE.

However, our empirical results across both discrete and continuous domains indicate that FairDICE maintains robust performance for sufficiently small values of $\beta$. To support this claim, we refer to the results in both domains. In the discrete setting (Random MOMDP, Figure 2), FairDICE consistently achieves high Nash Social Welfare across broad range of relatively small $\beta$ values (e.g., from 0.001 to 1). This suggests that finding a suitable $\beta$ is not a matter of delicate tuning. Similarly, in the continuous domain, as shown in Appendix H, FairDICE exhibits stable performance across a broad range of $\beta$ values.

While it is true that offline RL algorithms can be sensitive to hyperparameter choices, our findings suggest that FairDICE is relatively robust in this regard. We will revise the paper to clarify this point and provide additional guidance on selecting $\beta$ in practice.

[Q3: Robustness of FairDICE to Limited Data Quality and Coverage in Offline Multi-Objective RL]

As offline RL methods are often sensitive to the quality and coverage of the dataset, we provide empirical evidence that FairDICE exhibits a degree of robustness to suboptimal data quality and limited coverage.

Regarding data quality, as shown in Section 7, we evaluated FairDICE on both the expert and amateur datasets from the D4MORL benchmark [1], and it consistently achieved high Nash Social Welfare across both settings.

To further assess robustness to limited data coverage, we conducted additional experiments where we filtered out trajectories whose preference weights lie near the center of the simplex. Specifically, we removed trajectories in which all preference weights fall between 0.4 and 0.6. This filtering removes data points likely to represent balanced or fair trade-offs, resulting in a more challenging offline dataset.

The results are shown in the table above. Nash Social Welfare (Full) refers to performance on the original dataset without any filtering, while Nash Social Welfare (Filtered) reports performance after removing the trajectories. Each result shows the average Nash Social Welfare (NSW) over 5 seeds. FairDICE continued to perform reliably, demonstrating its ability to optimize fairness-driven objectives even under biased or sparse data conditions.

We will incorporate these additional robustness results into the updated version of the manuscript to better highlight FairDICE’s robustness.

References:

[1] Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via offline multi-objective rl. In The Eleventh International Conference on Learning Representations, 2023.

Dear Reviewer YvDZ,

We sincerely thank you for your detailed comments and insightful suggestions. We hope that our responses below provide the necessary clarification and help resolve the points you raised.

[W1: Clarification Needed on Domain and Interpretation of Concavity Assumption for Utility Function $u$]

To clarify, the concavity we refer to applies individually to each objective's return component, not over time steps. That is, the total objective is of the form $\sum_{i}u_{i}(R_{i})$, where each $u_{i}$ is a concave function applied to the return of the $i$-th objective.

We use concave functions to promote fairness across objectives. This approach is motivated by the principle of diminishing marginal utility, which implies that an identical gain in return provides more utility to underperforming objectives than to well-performing ones. As a result, the optimization naturally prioritizes weaker objectives, learning to more balanced and equitable outcomes.

This design choice is also supported by prior literature on fairness in multi-objective optimization, where per-objective concave functions or fairness-aware metrics (e.g., Nash Social Welfare) have been commonly used to balance outcomes [1, 2, 3]. We will revise the paper to clearly explain both the interpretation and the motivation behind this formulation.

[W2,Q2: Clarification Needed on the Importance of the Equivalence Between Regularized Linear MORL and Fair MORL]

The equivalence we establish serves a crucial theoretical and practical purpose: it demonstrates precisely why FairDICE offers a more principled and efficient approach compared to naive approaches.

Theoretically, the equivalence proves that for a given fairness objective (e.g. Nash Social welfare), there exists a single, specific set of preference weights that a regularized linear MORL agent would need to find the truly optimal fair policy.

Practically, however, finding these "magic" weights with a linear MORL method poses a major challenge. Since these optimal weights are unknown beforehand, one must perform brute-force grid search, a process that becomes computationally infeasible due to the curse of dimensionality as the number of objectives increases.

This is where the significance of FairDICE becomes clear. Instead of requiring this inefficient, extensive search, FairDICE treats the preference weights as optimization variables, directly learning the optimal weights as part of its end-to-end training process. In essence, it provides a principled way to find the optimal fair policy in a single run, without any weight sweeping.

Although this motivation is briefly mentioned in the final paragraph of Section 5, we agree that a clearer and more explicit explanation would help readers better understand this advantage. We will revise the text accordingly in the final version.

[W3,Q3: Why is FairMORL a single point in Figure 5? Is it possible to elicit different tradeoffs? If yes, where do they lie in this figure, and why is this not compared?]

We would like to clarify that the primary goal of FairDICE differs fundamentally from that of linear MORL methods like MORvSP. Linear MORL approaches aim to find a wide range of Pareto-optimal solutions, leaving the final decision of which trade-off is "best" to the user. In contrast, FairDICE, along with other non-linear fair MORL methods [1, 2, 3], is designed to directly find a single, principled solution that is optimal under a pre-defined, meaningful fairness criterion (e.g., Nash Social Welfare). Therefore, the single point in Figure 5 is a deliberate feature of our design, not a limitation.

In Figure 5, FairDICE appears as a single point because it optimizes a fixed nonlinear scalarization function: Nash Social Welfare (NSW), defined by $u_i(x) = \log x$. However, FairDICE allows for eliciting different trade-offs by changing the underlying utility function. For example, the widely used α-fairness formulation allows interpolation across a spectrum of fairness criteria:

u_i(x) = \\begin{cases} \\frac{x^{1-\\alpha}}{1 - \\alpha} & \\text{if } \\alpha \\ne 1 \\\\ \\log(x) & \\text{if } \\alpha = 1 \\end{cases}

Smaller values of $\alpha$ (e.g., $\alpha \to 0$) emphasize efficiency and resemble utilitarian welfare, whereas larger values (e.g., $\alpha \to \infty$) prioritize equality, approaching max-min fairness. In addition, a weighted Nash objective, such as $u_i(x) = w_i \log x$, can be used to reflect asymmetric priority across objectives. Each of these alternative formulations would yield a different point on or near the Pareto front.

While we did not compare multiple trade-offs in this figure, we agree that such an analysis could provide additional insight. We will clarify this design choice in the paper and discuss the potential to explore alternative fairness trade-offs as part of future work.

References:

[1] Zimeng Fan, Nianli Peng, Muhang Tian, and Brandon Fain. Welfare and fairness in multi-objective reinforcement learning. arXiv preprint arXiv:2212.01382, 2022.

[2] Giseung Park, Woohyeon Byeon, Seongmin Kim, Elad Havakuk, Amir Leshem, and Youngchul Sung. The max-min formulation of multi-objective reinforcement learning: From theory to a model-free algorithm. arXiv preprint arXiv:2406.07826, 2024.

[3] Mridul Agarwal, Vaneet Aggarwal, and Tian Lan. Multi-objective reinforcement learning with non-linear scalarization. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 9–17, 2022.

Dear Reviewer Rez3,

Thank you for your thoughtful and constructive feedback. We appreciate your careful review and hope that the following response addresses your questions and alleviates your concerns.

[W1: Clarification on Novelty Beyond Standard Convex Reformulation and DICE-RL]

To the best of our knowledge, introducing slack variables to derive an equivalent convex optimization problem is novel in the context of reinforcement learning. In typical formulations of Fair MORL, the objective takes the form of a concave utility function applied to an expected return, as in problem (P1). A direct approach to solving (P1) is challenging because the expectation is nested inside the non-linear (concave) function, which leads to biased sample-based estimation.

To address this challenge, we introduce slack variables to move the term $\sum_{s,a} d(s,a) r_i(s,a)$ outside the concave utility function $u_i$ (to linear constraints)​, resulting in a reformulated problem (P2). This crucial step enables us to develop a sample-based algorithm, FairDICE, which circumvents the bias issue incurred by taking expectation inside the concave function.

Although the derivation borrows elements from DICE-RL to account for distribution shift, simply applying DICE to a multi-objective setting is non-trivial—especially when fairness and optimality must be preserved across conflicting objectives. Our contribution lies in identifying how to retain these properties within a convex dual structure, which, to our knowledge, had not been done before in this context.

[W2,Q1: Lack of Explanation on Practical Optimization of FairDICE’s Bi-Level Objective with Convex Conjugate Term]

The use of a maximization in the definition of the convex conjugate may give the impression of a bi-level optimization. However, once the function $f(x)$ is specified, its conjugate $f^{*}(y)$ can be derived in closed form. For example, in our implementation of FairDICE (Appendix F), we use the soft-$\chi^2$ divergence as the regularization:

f(x) = \\begin{cases} x \\log x - x + 1, & x < 1 \\\\ 0.5(x - 1)^2, & x \\geq 1 \\end{cases} \\quad \\text{and} \\quad u(x) = \\log x.

The corresponding conjugate functions and $(f')^{-1}(x)$ are given analytically by:

f^{\\ast}(y) = \\begin{cases} e^y - 1, & y < 0 \\\\ 0.5y^2 + y, & \\text{otherwise} \\end{cases} \\quad \\text{and} \\quad u^*(y) = 1 + \\log y.

(f')^{-1}(x) = \\begin{cases} e^x, & x < 0 \\\\ 1+x, & \\text{otherwise} \\end{cases}

Therefore, the final objective can be computed directly and does not require solving an inner optimization loop.

The practical optimization procedure is provided in Algorithm 1 in Appendix F (p. 25). To improve clarity and accessibility, we will revise the main text to include a concrete example of the regularization function $f$, the utility function $u$, and their convex conjugates, and clearly reference the relevant section of the Appendix where the full algorithm is described.

[Q2: If the setting is online, can we design a practical algorithm based on the dual formulation (3)?]

Equation (3) by itself is not directly suitable for constructing a practical online algorithm, for two main reasons. First, the variable $d(s,a)$ does not generally correspond to the stationary distribution of any policy unless it satisfies the Bellman flow constraints. Second, even if we were to compute expectations with respect to $d(s,a)$, the $F_d(s)$ term depends on transition probabilities for arbitrary $(s,a)$, which are typically unknown in standard model-free settings (both online and offline). These challenges limit the direct applicability of (3) and motivate the subsequent derivations in our paper that transform it into a practical, sample-based objective.