Learning the Optimal Policy for Balancing Short-Term and Long-Term Rewards

Thank you for approving our work and for the helpful suggestions. Below, we address your concerns and questions.

W1: The experiment uses partial real data with synthetic generation of short-term and long-term rewards.

Response: Thanks for your comments. We fully agree that applying the proposed method to other domains, such as robotic planning and RL, would be beneficial.

However, we would like to clarify that our method focuses on policy learning in causal inference, which has peculiarities that cannot be correctly evaluated using real-world data. Here are the reasons:

In causal inference, the primary focus is on estimating treatment effects. Various methods achieve this using observed data under several identifiability assumptions. However, it is impossible to evaluate a method's performance based solely on observed data because the true treatment effects are always unknown in real-world data. Therefore, almost all experiments in causal inference [1-5] are conducted using synthetic data where the true treatment effects are known.

Next, we give a detailed explanation. Consider a binary treatment $A\in\{0,1\}$, covariates $X$, and outcome $Y$. Although $X$, $A$, and $Y$ are observed variables, they cannot alone define treatment effects. To define treatment effects, we denote $Y(a)$ as the potential outcome if $A$ were set to $a$ ($a$=0, 1). Thus, each individual has two potential outcomes: $Y(1)$ and $Y(0)$. The observed outcome $Y_i = A_i Y_i(1) + (1-A_i) Y_i(0)$, i.e., the observed outcome is the potential outcome corresponding to the received treatment. Then individualized treatment effect is defined as
$ITE_i=Y_i(1)-Y_i(0),$ which measures the treatment effect for individual $i$. Since each individual can only receive one treatment, we observe either $Y_i(0)$ or $Y_i(1)$, but not both. This is the fundamental problem of causal inference. Thus, we cannot obtain the true ITE from the observed data because we cannot simultaneously observe $Y_i(0)$ and $Y_i(1)$. In this article, we study the policy learning problem, which is also a causal problem. Consistent with the above analysis, due to the fundamental problem of causal inference, the rewards for a given policy cannot be evaluated using observed data alone.

W2: It seems that compared to linear weighting, the proposed method seeks more short-term reward but is not necessarily better in terms of long-term reward.

Response: Thanks for your insightful comments. In this article, we aim to develop a policy learning approach that balances both short- and long-term rewards, rather than focusing solely on maximizing either of them.

To evaluate the proposed method, we mainly use three metrics: S-REWARD, L-REWARD, and $\Delta W$. S-REWARD measures the short-term reward induced by the learned policy, L-REWARD measures the long-term reward, and $\Delta W$ measures the balanced welfare, indicating the policy's ability to trade off between multiple rewards. Clearly, to evaluate an approach that balances both short-term and long-term reward, $\Delta W$ is the most important metric.

In the experimental results, the proposed method may not always perform better on L-REWARD, but it shows superior performance for $\Delta W$. This indicates that the linear weighting approach does not achieve a good balance, as it pursues long-term rewards at the expense of short-term rewards. In contrast, our approach proves to be Pareto optimal, pursuing long-term rewards while also maximizing short-term rewards.

Q1: Do we necessarily need these separate definitions? Can we put everything as a long-term reward that can be missing sometimes?

Response: Thank you for raising this point. Yes, both long-term and short-term outcomes can be missing, and the proposed policy learning method remains applicable. However, this will bring extra difficulty in identifying and estimating long and short-term rewards. From a high-level perspective, the proposed policy learning approach consists of two steps:

• Step 1: policy evaluation, estimating the short and long-term rewards;
• Step 2: policy learning, solving optimization problem (3) based on the estimated rewards.

The missingness of short and long-term outcomes mainly affects Step 1, which involves identifying and estimating the rewards. Once the rewards are estimated, Step 2 focuses on learning the optimal policy through advanced optimization algorithms.

In Step 1, when both outcomes are missing, the estimation method of [6] may be invalid. Developing novel methods for identifying and estimating rewards in this context is an interesting direction, and we leave it for future work.

Q2: How are the weighting and preference vectors chosen?

Response: Thanks for your comments. Preference vectors (PVs) are used to quantify an individual's preference for different objectives. In our article, we randomly generate 10 unit PVs (see lines 270-272), ensuring the consistency and comparability of the preference measures. Each component of the PV represents the importance of the decision maker's preference for different objectives. Each PV is considered as a weight vector in the linear weighting method, see lines 275-276. In addition, we added experiments for different number of PVs, see response W5 to reviewer hVvL.

[1] Johansson et al. Learning representations for counterfactual inference, ICML 2016

[2] Shalit et al. Estimating individual treatment effect: generalization bounds and algorithms.ICML 2017

[3] Shi et al. Adapting neural networks for the estimation of treatment effects. NeurIPS 2019

[4] Yoon et al. GANITE: Estimation of individualized treatment effects using generative adversarial nets, ICLR 2018

[5] Bica et al. Estimating the effects of continuous-valued interventions using generative adversarial networks. NeurIPS 2020

[6] Wu et al. Policy Learning for Balancing Short-Term and Long-Term Rewards. arXiv 2024

Below, we hope to address your concerns and questions.

W1 No enough explanation/experiments to demonstrate that the proposed method is optimal.

W2: When some of the rewards are interrelated, the linear weighting method can only achieve a suboptimal solution. The claim may not be rigorous. More explanation and experiments are required.

Response: Thanks for your comments. Since W1 and W2 are similar, we response them together.

First, we explain why the proposed method is optimal. In Lemma 2 of Section 3.2 (lines 191-194), we show that the proposed method achieves Pareto optimality. Specifically, a Pareto optimal solution is obtained by solving the optimization problem (5) (lines 181-182) using an iterative gradient-based update rule. Lemma 2 demonstrates that this iterative rule converges to the Pareto optimal solutions.

Second, we explain why the linear weighting (LW) method is suboptimal.

• Previous studies [1-4] highlight the limitations of LW method in multi-objective optimization. LW method is restricted by assumptions of linearity and convexity, potentially ignoring complex interactions among objectives.
• When multiple objectives have nonlinear relationships, the Pareto set (the collection of all Pareto optimal solutions, see line 100) may not be convex. Due to its linear restriction, the LW method may not effectively identify Pareto optimal solutions within non-convex regions, leading to suboptimal solutions.
• As illustrated in Figure 2 of [5], when dealing with two objective functions, the LW method may struggle to identify Pareto-optimal solutions when the Pareto set is nonconvex.

Finally, we conducted an extra experiment to further illustrate the limitations of LW method and the optimality of our method. We consider two objectives $min_x F(x)=[f_1(x),f_2(x)]^T,$ $f_1(x)=1-\exp{(-\sum_{i=1}^d(x_d-\frac1{\sqrt{d}})^2)}, f_2(x)=1-\exp{(-\sum_{i=1}^d(x_d+\frac1{\sqrt{d}})^2)},$ where $x$ is a 20-dimensional covariates and $x_d$ is the $d$-th element of $x$. For the optimization problem above, we know the true Pareto set. For clarity, we present 10 uniformly selected Pareto solutions in Table 1 below, from which we can see that the Pareto set is concave rather than convex.

Table 1

Next, we use both the proposed method and the LW method to solve the above optimization problem. Here we select ten preference vectors (they also are the weight vectors in the LW method). The results are shown in Table 2 below, where $(f_{1, LW},f_{2, LW})$ and $(f_{1, our},f_{2,our})$ represents the solution obtained by the LW method and our method, respectively. we can see that the LW method only finds solutions at the endpoints of the Pareto set (i.e., in the convex part), while our method successfully locates all Pareto optimal solutions within the whole region. This indicates that the LW method cannot achieve the Pareto optimal solutions.

Table 2

W3: For Pareto optimality, is it possible that $\theta$ is not optimal for some $\mathcal{\bar V}$ but is optimal for the overall $\mathcal{\bar V}$?

Response: Thanks for your comments. We kindly remind the reviewer that there might be a misunderstanding regarding the definition of Pareto optimality. Below, we provide a detailed explanation. A solution is Pareto optimal if it is impossible to improve on objective without worsening other objectives. Typically, multi-objective problems have numerous Pareto optimal solutions, and all of them form the Pareto set. These solutions represent the best trade-offs among different objectives, rather than optimality for a specific single objective. Thus, it is possible that a Pareto optimal solution achieves the optimal solution in some objectives but not for others.

W4: Some mathematical symbols and proprietary terms in the paper are not explained clearly.

Response: Thanks for your comments and we apologize for any lack of clarity. Below, we provide a detailed explanation of the mathematical symbols:

• We define $e(x) \triangleq P(A=1|X =x)$ (see line 110).
• The MOP represents multi-objective optimization (see line 91).
• In line 171, $v$ is an $M$-dimensional vector in the positive real space $R_+^M$, where $M$ is the number of objectives.
• The Karush-Kuhn-Tucker (KKT) conditions are a set of necessary conditions that characterize the solutions to constrained optimization problems. The KKT conditions can be seen as a set of constraints derived from the Lagrangian function that must be met for a solution to be considered optimal in a constrained optimization problem.
• Both $d_{r_t}$ and $d_{t}$ represent the descent direction at $t$-th iteration, where the former is for seeking the initial solution, while the latter is for finding the Pareto optimal solution.

[1] Miettinen. Nonlinear multiobjective optimization, 2012

[2] Marler et al. The weighted sum method for multi-objective optimization: new insights, 2010

[3] Censor. Pareto optimality in multiobjective problems,1977

[4] Ngatchou et al. Pareto multi objective optimization, 2005

[5] Shim et al. Pareto‐based continuous evolutionary algorithms for multiobjective optimization, 2002

(Regarding the response to W5-W7, please see global review.)

We sincerely appreciate your comments and thank you for the helpful suggestions. Below, we hope to address your concerns and questions.

W1: - Only the linear weighting method is used as the baseline. I am wondering if there are any other methods that can be used for comparison. If not, why? Since both IHDP and JOBS are widely used.

Response: Thanks for your comments. From a high-level perspective, the proposed policy learning approach consists of the following two steps:

• Step 1: policy evaluation, estimating the short-term and long-term rewards $\mathcal{V}(\theta, s_i)$ and $\mathcal{V}(\theta, y_j)$;

• Step 2: policy learning, solving the optimization problem (3) based on the estimated values of $\mathcal{V}(\theta, s_i)$ and $\mathcal{V}(\theta, y_j)$.

For Step 1, the previous work [1] is well-established. However, for Step 2, it only uses a simple linear weighting method. In this article, we primarily focus on Step 2 and adopt the method of [1] in Step 1, given in Appendix A.

As discussed in the second paragraph of the Introduction (lines 32-35), the policy learning problem of balancing short and long-term rewards remains largely unexplored. There is only a single work [1] that addresses this problem, using a simple linear weighting method. Therefore, our work primarily compares with this approach.

In addition, we also tried to consider the epsilon constraint method, but it can be converted into a linear weighted method through certain transformations, so we did not choose it as an additional baseline.

Nevertheless, to make a more comprehensive comparison, we added two extra estimation methods in Step 1, and compare our proposed optimization method with the linear weighting method in Step 2. Specifically, the two extra estimators are given as

• OR estimator $\hat{\mathbb{V}}(\pi;s)^{OR}=\frac{1}{n}\sum_{i=1}^{n} [\pi(X_{i}) \hat{\mu}_{1}(X_i)+(1-\pi(X_i))\hat{\mu}_0(X_i)]$

$\hat{\mathbb{V}}(\pi;y)^{OR}=\frac{1}{n}\sum_{i=1}^{n}[\pi(X_{i}) \hat{\tilde{m}}_1(X_i,S_i)+(1-\pi(X_i)) \hat{\tilde{m}}_0(X_i,S_i)]$

• DR estimator $\hat{\mathbb{V}}(\pi;s)^{DR}=\frac{1}{n}\sum_{i=1}^{n}\Big[\pi(X_i)\left(\frac{A_i(S_i-\hat{\mu}_1(X_i)}{\hat{e}(X_i)}+\hat{\mu}_1(X_i)\right)+(1-\pi(X_i))\left(\frac{(1-A_i)(S_i-\hat{\mu}_0(X_i)}{1-\hat{e}(X_i)}+\hat{\mu}_0(X_i)\right)\Big]$

$\hat{\mathbb{V}}(\pi;y)^{DR}=\frac{1}{n}\sum_{i=1}^{n}\Big[\pi(X_i)\left(\frac{A_i(Y_i-\hat{\tilde{m}}_1(X_i,S_i)}{\hat{e}(X_i)}+\hat{\tilde{m}}_1(X_i,S_i)\right)+(1-\pi(X_i))\left(\frac{(1-A_i)(Y_i-\hat{\tilde{m}}_0(X_i,S_i)}{1-\hat{e}(X_i)}+\hat{\tilde{m}}_0(X_i,S_i)\right)\Big]$ The associated experimental results on JOBS are presented below.

From the experimental results, we can see that our method achieves better performance on the evaluation metric $\Delta{W}$ (the most important evaluation metric), and our method also performs better in terms of overall performance.

[1] Wu et al. Policy Learning for Balancing Short-Term and Long-Term Rewards. arXiv:2405.03329, 2024.

We sincerely appreciate your approval of the idea and the novelty of this work and thank you for the helpful suggestions. Below, we hope to address your concerns and questions.

W1: This paper proposes an important multi-objective optimization algorithm. But the title of this paper seems to be just a specific application scenario. In what other scenarios can this algorithm be applied?

Response: Thank you for the insightful comments. As the reviewer noted, we propose a general multi-objective optimization algorithm, making the proposed method applicable to various scenarios involving multi-objective balance. Balancing multiple objectives is particularly critical in the field of trustworthy AI, where the goal is not only to achieve desirable prediction performance but also to address other important aspects such as fairness [1,2] and non-harm [3,4].

W2: The experimental part is mainly conducted in a constructed environment, and it is unclear how difficult it is in the field of causal inference.

Response: Thanks for the constructive comments. In the field of causal inference, the primary focus is on estimating treatment effects. With several identifiability assumptions, various methods have been developed to achieve this goal using the observed data only. However, it is impossible to evaluate a method's performance based solely on the observed data. The basic rationale is that the true treatment effects are always unknown in real-world data. Thus, the experiments in almost all the articles in causal inference [5-9] are conducted in a constructed environment where the true treatment effects are known.

Next, we give a detailed explanation. Consider the case of a binary treatment variable $A\in\{0,1\}$, $X$ is covariates, and $Y$ is the outcome. Although $X$, $A$, and $Y$ are all observed variables, they cannot alone define treatment effects. To define treatment effects, we denote $Y(a)$ as the potential outcome that would be observed if $A$ were set to $a$ ($a$=0, 1). Thus, each individual has two potential outcomes: one is $Y(1)$ if the individual receives the treatment ($A=1$), and the other is $Y(0)$ if the individual don't receives the treatment ($A=0$). The observed outcome $Y_i = A_i Y_i(1) + (1-A_i) Y_i(0)$, that is, the observed outcome is the potential outcome corresponding to the received treatment. With the notation of potential outcomes, we define the individualized treatment effect as $ITE_i=Y_i(1)-Y_i(0),$ which measures the magnitude of the treatment effect for individual $i$. Since each individual can only receive one treatment, we observe either $Y_i(0)$ or $Y_i(1)$, but not both. This is known as the fundamental problem of causal inference [10]. As a result, we cannot obtain the true ITE from the observed data because we cannot simultaneously observe $Y_i(0)$ and $Y_i(1)$.

In this article, we aim to find the optimal policy that maximizes rewards $V(\pi)=E[\pi(X)Y(1)+(1-\pi(X))Y(0)]$, which is also a causal problem. Consistent with the above analysis, due to the fundamental problem of causal inference, for a given policy $\pi$, the rewards cannot be evaluated with the observed data only.

W3: The v in line 171 is missing \bar. In Appendix B, t in line 5 of Algorithm 1 should start from 0.

Response: Thank you for pointing this out. We will revise it accordingly.

Q1: How to deal with a situation where long-term rewards are missing? Should the data be ignored when solving the network?

Response: Thank you for raising this point and we apologize for the lack of clarity. From a high-level perspective, the proposed policy learning approach consists of the following two steps:

• Step 1: policy evaluation, estimating the short-term and long-term rewards $\mathcal{V}(\theta, s_i)$ and $\mathcal{V}(\theta, y_j)$ for a given policy;
• Step 2: policy learning, solving the optimization problem (3) based on the estimated values of $\mathcal{V}(\theta, s_i)$ and $\mathcal{V}(\theta, y_j)$.

The missingness of long-term outcomes primarily affects Step 1, which involves the the identifiability and estimation of $\mathcal{V}(\theta, s_i)$ and $\mathcal{V}(\theta, y_j)$ (see line 81 of the manuscript). Once $\mathcal{V}(\theta, s_i)$ and $\mathcal{V}(\theta, y_j)$ are estimated, Step 2 focuses on learning the optimal policy by developing advanced optimization algorithms.

For Step 1, the previous work [11] is well-established. However, it does not address Step 2, as it only uses a simple linear weighting method. In this article, we primarily focus on Step 2 by proposing a novel optimization algorithm. Additionally, we establish connections with other methods (e.g., $\epsilon$-constraint) to guide the selection and provide interpretation of the preference vectors. For Step 1, we adopt the method of [11] and the associated results are presented in Appendix A.

References

[1] Kusner et al. Counterfactual fairness. NeurIPS 2017

[2] Ethan et al., Fairness-Oriented Learning for Optimal Individualized Treatment, JASA 2023

[3] Kallus N., Treatment Effect Risk: Bounds and Inference, FAccT 2022

[4] Li et al., Trustworthy Policy Learning under the Counterfactual No-Harm Criterion, ICML 2023

[5] Johansson et al. Learning representations for counterfactual inference, ICML 2016

[6] Shalit et al. Estimating individual treatment effect: generalization bounds and algorithms. ICML 2017

[7] Shi et al. Adapting neural networks for the estimation of treatment effects. NeurIPS 2019

[8] Yoon et al. GANITE: Estimation of individualized treatment effects using generative adversarial nets, ICLR 2018

[9] Bica et al. Estimating the effects of continuous-valued interventions using generative adversarial networks. NeurIPS, 2020.

[10] Holland. Statistics and causal inference. JASA 1986

[11] Wu et al. Policy Learning for Balancing Short-Term and Long-Term Rewards. arXiv 2024