Beyond Expected Returns: A Policy Gradient Algorithm for Cumulative Prospect Theoretic Reinforcement Learning

• CPT and RLHF are not mutually exclusive. While CPT and trajectory-based RL (say e.g. RLHF) both offer frameworks for incorporating human preferences into decision making, we would like to highlight that CPT and RLHF are not mutually exclusive. We can for instance use CPT to design an initial reward structure reflecting human biases, then refine it with RLHF. We can also consider to further relax the requirement of sum of rewards (which already has several applications on its own) and think about incorporating CPT features to RLHF. Some recent efforts in the literature in this direction that we mentioned in our paper include the work of Ethayarajh et al. (ICML 2024) ‘Model alignment as prospect theoretic optimization’ which combines prospect theory with RLHF (without probability weight distortion though, which limits its power). Note that the ideas of utility transformation and probability weighting are not crucially dependent on the sum of rewards structure and can also be applied to trajectory-based rewards or trajectory frequencies for instance. We believe this direction deserves further research, one interesting point would be how to incorporate risk awareness from human behavior to such RLHF models using ideas from CPT.

Thank you for the interesting question. We have now incorporated this discussion to appendix D (p. 18, in the revised manuscript) due to space constraints.

Another weakness is related to the empirical study of the paper. The experiments only demonstrate the effectiveness of the proposed algorithm but do not include those justifying the necessity of the CPT problem. It would be much better if the authors could demonstrate that in some applications, the data shows that human behavior can be accurately predicted with CPT but not other theories.

About the importance of our problem formulation and its empirical relevance. While our empirical results do not focus on the importance of our CPT problem formulation and mainly serve an illustrative purpose given our theoretical and algorithmic contributions to CPT-RL, we would like to stress that CPT has been tested and effectively used in a large number of compelling behavioral studies that we cannot hope to give justice to here. It is far from being an intellectual theoretical curiosity disconnected from practice. Besides the initial findings of Tversky and Kahneman for which the latter won the Nobel Prize in economics in 2002, please see a few recent references below for a broad spectrum of real-world applications ranging from economics to transport, security and energy, mostly in the stateless (static) setting, also reflecting how active and impactful this research is in other fields. We quote some of the results for each paper for the convenience of the reader to reply to the reviewer’s concern. We hope these examples convince the reviewer of the importance of the problem we address which is an extension to the dynamical setting via our RL formulation for sequential decision making.

1. Rieger, M. O., Wang, M., & Hens, T. (2017). Estimating cumulative prospect theory parameters from an international survey. Theory and Decision, 82, 567-596. ‘We conduct a standardized survey on risk preferences in 53 countries worldwide and estimate cumulative prospect theory parameters from the data. The parameter estimates show that significant differences on the cross-country level are to some extent robust and related to economic and cultural differences. In particular, a closer look on probability weighting underlines gender differences, economic effects, and cultural impact on probability weighting.’ Note here the explainability feature of CPT that we highlighted above in our discussion.

2. Ebrahimigharehbaghi, S., Qian, Q. K., de Vries, G., & Visscher, H. J. (2022). Application of cumulative prospect theory in understanding energy retrofit decision: A study of homeowners in the Netherlands. Energy and Buildings, 261, 111958. ‘CPT correctly predicts the decisions of 86% of homeowners to renovate their homes to be energy efficient or not. EUT, on the other hand, overestimates the number of decisions to renovate: it incorrectly predicts retrofit for 52% of homeowners who did not renovate for energy efficiency reasons. Using the estimated parameters of CPT, the cognitive biases of reference dependence, loss aversion, diminishing sensitivity, and probability weighting can be clearly seen for different target groups.’

We thank the reviewer for their time, for their valuable feedback and thoughtful comments. We appreciate that the reviewer is faithfully and accurately reporting our contributions. We provide a detailed discussion below regarding the reviewer’s concerns about the importance of the problem and its relevance in comparison to other approaches such as trajectory-based RL and RLHF in particular.

While I appreciate the contributions the authors made to CPT-based RL problems, I am not convinced of the importance of CPT-based RL problems by reading this paper. The paper cites another paper "... this is particularly important in applications directly involving humans in the loop such as e-commerce, crowdsourcing and recommendation to name a few" to argue that this class of problems has important applications.

CPT is a popular model in behavioral economics originating from economics and psychology that was recognized by a Nobel prize attributed to Daniel Kahneman in 2002. We believe that developing a paradigm extending CPT to sequential decision making is an important and natural extension as it allows to broaden the scope of applications and factor in its main features by taking into account the subjective valuation of outcomes (utility) and the subjective weighting of probabilities. CPT-based RL unifies different settings including in particular risk-sensitive RL that has by now endless applications in RL. Please see our response below to your last comment for a detailed discussion regarding the importance of our problem and existing applications. We thank the reviewer for their comment, we will certainly add further motivation in the introduction to further support the importance of our problem. We have now added section C in the appendix p. 17 (due to space constraints) to discuss applications along the lines of the last comment below.

However, it is not clear from the cited paper whether other formulations that take into account human behavior can also handle these applications well. In fact, compared with CPT, I feel that a more principled way that considers human behavior is trajectory-based reward RL problems (one reward for the entire trajectory) with human preferences, like RLHF in large language models. While CPT assumes a structure of the final reward (the final metric to optimize involves a weight function, a utility function, and a summation of rewards), these weight functions are unknown and must be learned from data. In contrast, in trajectory-based reward RL problems, there is no assumption on the structure of the metric being optimized. And the metric is learned with human preference data. I wonder how authors think about the pros and cons of CPT in terms of its applications compared to trajectory-based reward RL problem settings.

First of all, we do not exclude that other approaches might also be useful for modeling human behavior (and there are others as you mention), this is an open and promising research area that we believe has yet many interesting research directions to offer. We also do not claim that our approach is the unique best way to tackle the problem. Nevertheless, we do believe it is a principled approach rooted in an established literature in behavioral economics that would gain to be developed for sequential decision making. Our paper contributes to this effort by proposing a practical PG algorithm. While the CPT approach has not yet been well-established in the machine learning community, we believe it has interesting features to offer and it is already pervasive in RL through its particular cases risk-sensitive and safe RL. We provide a more detailed discussion below regarding these aspects and comment on your interesting question regarding the comparison with other paradigms such as RLHF/trajectory-based reward RL and the pros and cons of CPT.

Please see the rest of our response to your above comment in what follows.

Your feedback on the usefulness of CPT-RL is appreciated. One suggestion is to make the example in your paper more concrete. Specifically, in the patient treatment example, you explained the potential benefits of weight functions, utility functions, and reference points. However, it would be more convincing if you could reference a study, if available, that models these functions and points using real patient treatment data.

I have another question regarding the extension of algorithms beyond Policy Gradient (PG) to the CPT setting. Specifically, when it comes to learning a value function, do you think there are straightforward ways to adapt RL algorithms like TD learning or Q-learning to the CPT framework?

Lastly, I'm a bit confused about your claim that "several risk measures are also particular cases of CPT values: Variance, Conditional Value at Risk (CVaR), distortion risk measures, to name a few." You also mentioned in the caption of Table 1 that "w+ and w− are often required to be continuous, which would exclude VaR and CVaR." These two statements seem conflicting—one suggests that CPT encompasses CVaR, while the other implies it does not. Could you clarify this?

The policy gradient calculation depends on the property that the function u is non-decreasing. Can we work with other u where non-decreasing is not guaranteed?

• Strict monotonicity is needed in some of the proofs of our results, mainly Theorem 4, see e.g. proof of 1 implies 2 in p. 24 and proof of 5 implies 2 in p. 22. However, it is not formally required to derive our policy gradient theorem. Note also that even Proposition 7 for estimating the integrals does not require it as soon as $\xi_{i/n}^{+,-}$ denote the right quantiles as we defined them in the proposition. See point 3 below for further comments.

• We focus on the monotone setting because typical human behavior tends to prefer better outcomes over worst ones, e.g. higher gains over smaller ones and smaller losses over larger losses. This is captured by the monotonicity assumption on the utility function: utility increases with increasing gains and decreases with increasing losses. This is a fundamental assumption in economics and decision theory which is consistent with how humans evaluate outcomes. Nevertheless, there are cases where this assumption might not hold if this is what the reviewer is referring to.

• Mere monotonicity is mainly useful in our algorithm to guarantee that the utility values are sorted in the same order as the returns obtained from the sampled trajectories (see step 6 of the algorithm for quantile estimation computation). This leads to a simple computation of the quantiles of the utilities: Once the returns are sorted, the utilities are also sorted in the same way and quantiles can be read from the sorted list. If monotonicity does not hold anymore, one has to be more careful about this computation and adapt step 6 of the algorithm accordingly by simply computing the quantiles of the utilities using the right sorting of these (which would be different from the sorting of the returns). In that case, sorting the returns in step 5 is not needed and one only needs to sort the utilities (utility function applied to the returns). Apart from this technical detail, we do not foresee any major impediment to using the algorithm without the strict monotonicity assumption.

• We thank the reviewer for this question, we will add a remark to the paper accordingly.

Thank you for your review, please let us know if you have any further concern or questions.

We thank the reviewer for their time and feedback. We answer their questions and reply to their comments in the following.

The algorithm is only validated in a few small domains, making it difficult to assess the performance of this policy gradient in more complex tasks.

Besides our small scale grid world experiments, please note that we did also test the performance of our PG algorithm on a continuous state action space setting with our electricity management scenario. In practice, prior work in CPT-RL (L.A. et al. 2016, ICML) did only consider a SPSA (zeroth order) algorithm (which we also compared to) in a traffic signal control application on a 2x2-grid network. Our PG algorithm has clearly an advantage compared to a zeroth order algorithm which only relies on function values and which is harder to scale to larger state action spaces. We have now performed additional simulations during the rebuttal phase for a financial trading application (see response to reviewer Ydtj, results and precise description in the last two pages of our revised manuscript, all modifications in blue).

The computation of the integral for the CPT value appears complex and time-consuming due to the use of Monte Carlo sampling.

Monte Carlo sampling is used even for the vanilla policy gradient algorithm (say Reinforce) for standard expected return RL. We use a similar procedure here which is simple. Our additional required quantile estimation procedure requires a mild sorting step which can be executed in $O(n \log n)$ running time (without even invoking parallel implementations) where $n$ is the length of the rewards to be sorted.

Suggestion: The authors may consider demonstrating the benefits of using CPT over standard risk-neutral RL in a concrete reinforcement learning problem at the beginning of the paper to help readers better understand CPT.

We have provided concrete simple examples of CPT in the introduction to give the reader some intuition without introducing any notation nor background. For clarity, we prefer to defer a detailed exposition of examples of CPT-RL to later in the paper (simulation section) once the problem formulation is precisely exposed and explained. Thank you for the suggestion, we will add a pointer to the introduction to directly refer the reader to the relevant section for an example.

This paper considers the problem in a non-discounted setting, i.e., γ=1 as shown in Line 181. The claim in line 261 does not apply to the discounted case for exponential utilities. Discussion may be needed. see [Entropic Risk Optimization in Discounted MDPs, AISTATS 2023].

• We thank the reviewer for their comment and for the reference. Indeed, we focus on the undiscounted setting in our work. This is precisely because some of the issues discussed in the paper you mention arise when considering discounted rewards (e.g. lack of positive homogeneity). The entropic risk measure (ERM) mentioned in the reference you provide seems to be actually rather consistent with our result (Theorem 4), up to the monotone log function applied to the objective (which does not fundamentally change the policy optimization problem). Comparing (5) and (8) in the reference with our (EUT-PO) problem formulation, ERM is rather consistent with the exponential form we provide in Theorem 4 (up to the fact that our theorem does not apply to the discounted setting as we currently state it). As for the entropic value-at-risk (which builds on the entropic risk measure), it seems that it is not exactly an instance of our problem formulation (EUT-PO) because of the supremum over $\beta$ transformation coupling ERM with the additive log term (besides the discounted setting).

• Although discounting is widely adopted in the RL community, especially for infinite horizon settings, we do not find the finite horizon to be a restrictive setting in practice in applications. As for the extension to the infinite horizon, we find the use of positive quasi-homogeneity to adapt the proof and show the existence of an optimal deterministic Markov policy (which is crucially non-stationary) for the special case of Entropic Risk Measure (Theorem 3.3 therein) interesting. This observation might actually give some hope to extend our result to the discounted setting using similar techniques. We will add a discussion regarding this interesting point, thanks again for your comment and the useful reference that we were not aware of.

We thank the reviewer for their feedback. We address their concerns in the following. Regarding the importance of CPT-RL, we refer the reviewer to our detailed response to reviewer aLSN and appendix C p. 17 which we added to the manuscript. We have also augmented the introduction with l. 63-68 to highlight applications. Concerning examples motivating CPT-RL, we provide another example below and we have now added it to the main part of the paper in section 2 p. 4-5 to illustrate our CPT-RL problem formulation as suggested by the reviewer (deferring related work to the appendix due to space constraints). As for the utility function choice, we will add a discussion following our response to your question above.

Additional concrete example of CPT-RL in healthcare: Personalized Treatment for Pain Management. CPT is not just a generalization, its features are important in applications, especially when human perception and behavior matter. Existing traditional RL approaches often lack a behavioral perspective and CPT-RL allows for more empathetic and realistic sequential decision making. Here is another concrete example to illustrate the importance of CPT-RL and its differences compared to risk sensitive RL to provide more intuition to the reader.

-Scenario: The goal is to help a physician manage a patient's chronic pain by suggesting a personalized treatment plan over time. The challenge here is to balance pain relief and the risk of opioid dependency or other side effects that might be due to the treatment, i.e. short-term relief and longer term risks.

-Our approach: We propose to train a CPT-RL agent to help the physician.

Why sequential decision making?

(a) The physician needs to adjust treatment at each time step depending on the patient’s reported pain level as well as the observed side effects. Note here that this is relevant to dynamic treatment regimes in general (such as for chronic diseases) in which considering delayed effects of treatments is also important (and RL does account for such effects). We refer the reader to section IV of Yu and Liu 2020, ‘Reinforcement Learning in Healthcare: A Survey.’ ACM Computing Surveys.

(b) Decisions clearly impact the patient's immediate pain relief, dependency risks in the future and their overall health condition.

Why CPT? Patients and clinicians make decisions influenced by psychological biases. We illustrate the importance of each one of the three features of CPT as introduced in our paper in section 2 (reference point, utility and probability distortion weight functions) via this example:

(a) Reference points: Patients assess and report pain levels according to their subjective (psychologically biased) baseline. Incorporating reference point dependence leads to a more realistic model of human decision-making as this allows for capturing e.g. expectations, past experience as well as their desired outcomes to define their perceived gains and losses. In our example, reducing pain from a level of 7 to 5 is not perceived the same way if the reference point of the patient is 3 of it is 5. In contrast, risk-sensitive RL treats every pain reduction as a uniform gain, regardless of the patient’s starting reference pain level.

(b) Utility transformation: Patients might often show a loss averse behavior, i.e., they might perceive pain increase or withdrawal symptoms as worse than equivalent gains in pain relief. Note here that loss aversion should not be confused with risk aversion (see definition and discussion in Schmidt and Zank 2005. ‘What is loss aversion?’ The Journal of Risk and Uncertainty.) In short, loss aversion can be defined as a cognitive bias in which the emotional impact of a loss is more intense than the satisfaction derived from an equivalent gain. For instance, in our example, a 2-point increase in pain might be seen as much worse than a 2-point reduction even if the change is the same in absolute value. This loss aversion concept is a cornerstone of Kahneman and Tversky’s theory. In contrast, risk aversion rather refers to the rational behavior of undervaluing an uncertain outcome compared to its expected value. Risk sensitive approaches might be less adaptive to a patient’s subjective preferences if they deviate from objective risk assessments.

(c) Probability weighting: Low probability events such as severe side effects (e.g., opioid overdose or dependency) might be overweighted or underweighted based on the patient's psychology.

Environment and transitions. A state is a vector of three coordinates (current pain level, dependency risk, side effect severity). Actions are treatments, e.g. no treatment, alternative treatment or opioid treatment. An episode ends if the patient develops full dependency or if pain is effectively managed.

CPT-RL vs Risk-averse RL: comparison. In terms of policies, risk-averse RL would favor non-opioid treatments unless extreme pain levels make opioids justifiable. In contrast, CPT-RL policies would prescribe opioids if pain significantly exceeds the patient’s reference point. As dependency risk increases, CPT-RL policies would transition to non-opioid treatments as a consequence of overweighting the probability of rare catastrophic outcomes. Notably, CPT-RL policies can oscillate between risk-seeking (to address high pain) and risk-averse (to avoid severe side effects). In contrast, a risk-sensitive agent focuses on minimizing variability in health states and dependency risks and would likely avoid opioids in most cases unless pain levels become extreme. Such risk-sensitive policies favor stable strategies (e.g., consistent non-opioid use), prioritizing low variance in patient outcomes.

Experiments. In our simulations, we have focused on examples emphasizing the behavioral economics motivation of CPT-RL. As for Mujoco, we have now also added an example on the inverted pendulum environment to demonstrate that our PG algorithm can be readily used in such environments as well (see page 38 in the appendix).

Limited applicability: The experiments did not demonstrate the advantages of using the proposed algorithm. It is unclear what the benefits of using CPT learning are over risk-sensitive RL or distributional RL. Although CPT is theoretically valuable, the empirical advantages of CPT over other risk-sensitive measures remain ambiguous in the presented results. The authors may consider including direct comparisons with risk-sensitive RL or distributional RL methods on the same tasks.

Our general PG theorem result and our novel proposed algorithm expand the applicability of CPT-RL in practice besides being theoretically grounded. We would like to highlight that our PG theorem unifies several settings including standard RL, risk sensitive, risk seeking, probability distorted settings and beyond under the umbrella of CPT-RL. Our algorithm which stems from such a general result is therefore general as for the different problem settings it can address. We refer the reader to p. 17 for a diagram representing our framework in the literature and Table 1 for different special case examples.

Advantage and applicability. We devoted a specific part of our experiments to clearly illustrate the scalability advantage of our PG algorithm compared to the zeroth order algorithm (please see sec. 5. (b) and fig. 3, the performance of the 0th order algorithm gets even worse with an even larger scale). We also demonstrated its applicability to different settings including the case of continuous state action settings, different utility functions including risk seeking and risk averse ones, and different probability weight functions as well.

Comparison to Risk sensitive RL. Risk sensitive RL is a particular case of our framework and our goal is not to outperform existing algorithms for specific tasks using our general algorithm. We also provided several examples where we use risk-sensitive measures though. We also note that prior work in risk-sensitive RL (e.g. Vijayan and L. A. AISTATS 2023 that we mention in related works) has also used similar SPSA algorithms as the one we compare to in section 5. (a). Moreover, particularizing our problem and algorithm to the special case of the risk-sensitive setting with exponential criteria results in an algorithm that bears similarity to existing work and PG algorithms such as the work of Noorani et al. 2023 and we refer the reader to this paper for comparisons to other risk-sensitive algorithms for that particular task. Distributional RL focuses on modeling and computing the entire distribution of returns. While distributional RL allows handling risk explicitly by approximating the distribution of returns, it is in general computationally demanding and it does not model subjective risk handling via utility and probability distortions like CPT.

Comparison to CPT.

• CVar or VaR in risk sensitive RL do not take into account transformed utilities (modeling perceived/subjective gain and loss values differently and wrt reference points) whereas exponential risk sensitive RL does not account for probability weighting (which models low and high probability events subjective perception). To appreciate this, please see Table 1 in p.17 and our experiments where we compare the different examples (please see figure 2).

• When an agent perceives a return differently from the raw return, then this is also modeled in our CPT framework and it is clear that CPT is more suitable than existing risk sensitive approaches. This is the case when (a) it has a different reference point regarding their perceived value, (b) its utility is not linear in the return, it is rather concave for gains and convex for losses (wrt their reference).

• Most importantly, we stress that our CPT framework captures several existing risk sensitive measures as particular cases and offers the flexibility to the decision maker to design their own. In particular, CPT can at the same time overweight low probability events and underweight highly probable ones with different magnitudes. Therefore, CPT allows to handle losses and gains separately. A similar flexibility is offered for utility functions. See figure 6 p. 18 for illustrations.

We thank the reviewer for their feedback. Please find below a point by point reply to your comments and concerns.

'The experimental section provides limited insight due to its use of relatively straightforward examples (traffic control and electricity management), which may not fully illustrate the complexities that arise in more realistic, high-stakes environments, such as finance or healthcare.'

Traffic control and electricity management are major high-stakes and complex applications which are extremely active research areas. Such applications need no motivation regarding their complexity and relevance in the real world. As a matter of fact a quick keyword search on app.dimensions.ai shows more than 200 000 (resp. more than 100 000) publications around traffic control (resp. electricity management) in 2023.

Our simulations. While our simulations rely on simplified models that cannot capture all the real-world intricacies of such complex applications, our goal is to illustrate how our problem formulation can be useful to solve concrete problems by giving a flavor about the scenarios one might consider in practice. We purposefully designed them to go beyond the standard vanilla gridworld that we have also considered. In particular, please note that we are using real-world public data available online for our simple electricity management application (please see appendix F. 6, Figure 19). We report electricity prices on a typical day on the market together with the total electricity production.

Besides our theoretical contributions which we would like to highlight, please let us briefly recall the purpose of each one of the applications we consider and the insights we provide: (a) in our traffic control application, we show the influence of the probability distortion function, (b) we illustrate the scalability of our PG algorithm to larger state spaces compared to the existing zeroth order algorithm in a grid environment with increasing state space size and (c) we test our algorithm in a continuous state-action space setting in our electricity management application.

Additional simulations. Finance and healthcare are also important applications. We thank the reviewer for mentioning these. Exploring all these applications in depth is beyond the scope of this paper. Nevertheless, during the rebuttal period we have now performed additional simulations in a financial application as requested by the reviewer. The goal is to train RL trading agents using our general PG algorithm in the setting of our CPT-RL framework. We use a Gym Trading environment available online using data from the Bitcoin USD (BTC-USD) market between May 15th 2018 and March 1st 2022. One can easily use any other real-world publicly available stock market data by providing a dataframe as input to build a corresponding RL environment. Our BTC-USD data is of the same kind as any historical data which can be found online (see e.g. https://finance.yahoo.com/quote/BTC-USD/history). The RL trading agent can take three classical positions (SHORT, OUT and LONG). We refer the reader to the last pages of our revised manuscript (modifications all in blue) for the figure as well as more details regarding the experiment. We hope these additional simulations further convince the reviewer about the practicality of our algorithm.

In Section 5(a), the observation that different weight functions lead to different policies in the grid environment could be strengthened by assessing these policies against risk measures (such as CVaR) beyond expected return. Without this, it is unclear if the CPT-RL-PO policy outperforms standard RL-PO under any specific risk-sensitive criteria.

Please note that we do consider a risk averse probability weight function to compare to the risk neutral case. We refer the reader to Table 2 p. 28, appendix F. 5 p. 31 for an illustration of the probability weight function w^+ used and Figure 18 for representations of the policies obtained that illustrate that the output policies solving our PO problem are meaningful compared to standard RL-PO.

The grid environment results are overly simplified and provide little substantive information. It would be beneficial to evaluate whether the derived policies' performance aligns meaningfully with their respective objectives.

• Our experiment with a varying grid world state space size illustrates the better scalability of our PG algorithm compared to the previously known zeroth order algorithm (see Figure 3).
• We did investigate the policies that we obtain by solving our problem using our novel PG algorithm, see appendix F.4, figure 16 p. 31 for representations of the policies we obtain and how they are consistent with the objectives set. See also figure 18 p. 32.
• We have now performed additional simulations to apply our methodology and use our PG algorithm in financial trading (see last part of the revised manuscript for a detailed exposition).