Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning

We kindly thank reviewer SKG7 for their review of our work. In the following, we answer the brought forward criticism and questions.

$\varepsilon$ threshold sensitivity

Reviewer rq81 is concerned about the sensitivity of our method with respect to manually selected thresholds.

Our method is indeed dependent on the manually selected $\varepsilon_i$ thresholds, however, this is the case for all $\varepsilon$-lexicographic MORL methods, since these thresholds are part of the problem definition. In the limitation section of our paper, we suggest multiple approaches to finding adequate values for $\varepsilon_i$ and note these are fundamentally problem-dependent, the same way that a reward function is.

See also our answer "Threshold ablation" to reviewer tsYf. We have included an additional ablation study with different $\varepsilon_i$ values in the revised manuscript (see Appendix H.3), to further illustrate how these scalars affect performance and agent behavior. We further note that we describe how adequate $\varepsilon_i$ thresholds can be selected in Section 6 of the paper.

Comparison to existing lexicographic RL algorithms

Reviewer rq81 suggests additional comparison with lexicographic RL algorithms.

As we write in the related work section, we believe to contribute the first algorithm for continuous action-space lexicographic MORL problems. As stated in our answer "8. Discrete PSQD version" to reviewer SKG7, a discrete version of our algorithm would be conceptually similar to existing works and we do not consider this comparison informative for learning in continuous spaces.

1. Relationship between Eq.(1) and Eq. (7)

Equation (1) defines a general lexicographic constraint and can also be found in related works on lexicographic RL. Equation (1) simply states that the optimization of subtask $i$ is constrained to a set $\Pi$ of policies, where all policies in $\Pi$ are also (near-) optimal to all higher-priority subtasks ${1, \dots, i-1}$.

Since the explicit computation of $\Pi$ is not practical, especially for probabilistic policies with continuous action spaces, in Eq. (7) we instead make a state-based version of Eq. (1), where the performance measures $J_i$ are the Q-functions. This is also consistent with existing related work on lexicographic RL.

We hope this clarifies the relationship of these two equations.

2. State-dependent $\varepsilon_i$

It's worth pondering whether εi should be state-dependent.

This is indeed an interesting discussion and an interesting research topic.

However, we want to highlight that, although $\varepsilon_i$ are fixed scalars, the Q-functions to which these thresholds relate are very much state-dependent. In effect, this results in indifference spaces that are very large (i.e. permissive) in some states, while only being restrictive in states where it really matters. For example, when far from the obstacle, the lexicographic constraint on obstacle avoidance allows nearly every action, however when close to the obstacle, it forbids the selection of those actions that would lead to obstacle collision (since those actions are clearly sub-optimal for the obstacle-avoidance task).

This can also be seen in Figure 7 in the appendix, where we visualize the agent and the indifference space at multiple locations in the environment. Most of these are relatively close to the obstacle, nevertheless, the indifference space in Subfigure 7.h is much larger and more permissive than in Subfigure 7.d or 7.f, due to varying proximity between the agent and the obstacle.

We hope this addresses reviewer rq81's questions regarding our work and again thank them for their review.

We kindly thank reviewer SKG7 for their review of our work. In the following, we answer the brought forward criticism and questions.

1. & 2.

We fully agree with reviewer SKG7 that pseudocode and pictograms will benefit understandability.

Therefore, in the revised version of the manuscript in Appendix "D PSQD algorithm details", we have added algorithm pseudocode for the pre-training step, the adaptation step, a pictographic overview of our framework (Fig. 5), and additional details regarding our method. We will release our full codebase once the paper is accepted, which will benefit the understandability and reproducibility of our work.

3

• 3.1 We believe lexicographic MORL has numerous benefits over traditional MORL which relies on scalarization: Lexicographic MORL algorithms allow for the transfer of simple subtask agents to complex lexicographic problems, benefits interpretability by decomposition, and offer constraint-satisfaction guarantees that can make for a safe exploration framework. Our method is for continuous action-space MDPs, unlike existing lexicographic RL methods for discrete action-space MDPs. We make an important, fundamental contribution by extending lexicographic MORL to continuous action-space MDPs, which have many applications in real-world problems, like robot control.

• 3.2 Lexicographic MORL can be seen as a special form of constrained MORL, where the constraints are of a lexicographic nature and for a chain-like directed, acyclic graph. This is not the same as, e.g. constrained policy optimization (Achiam et al, 2017), which places multiple constraints on the policy that are not lexicographically ordered. Furthermore, in lexicographic MORL, the constraints are based on the learned subtask solutions (e.g. Q-functions) and not based on manually defined functions, as is usually the case in constrained MORL.

• 3.3 The RL practitioner/designer decides on priority order. Priority order is part of the problem definition, in the same way that defining the MDP (state-space, action-space, reward function) is. Defining a lexicographic RL problem requires a priority order, in the same way that defining a regular RL problem requires a reward function. If the RL practitioner can not decide the order of two subtasks, it is possible to treat those subtasks with equal priority by simply summing those corresponding reward functions. We discuss this more in our response below to question 3.5.

• 3.4 We have experiments with three subtasks in Appendix G.2. These experiments show that changing the priority order changes the resulting behavior, which is the expected result. Changing the priority order changes the problem definition, therefore comparing the performance of multiple such agents does not make sense. Related to this is also our new ablation study on the priority thresholds in Section H.3 in the appendix.

• 3.5 If tasks are of equal importance, then it is valid to sum their reward functions. We can then learn a single Q-function $Q_{1+2}$ for $r_1 + r_2$. We can use $Q_{1+2}$ like any subtask Q-function and define a corresponding $\varepsilon_{1+2}$ threshold for this Q-function. Keep in mind, however, that simply adding the subtask reward functions can produce unexpected behavior, especially when these subtasks Q-functions are semantically incompatible. We argue that practitioners should consistently order tasks by priority. If the tasks are compatible, then the lexicographic constraints do not hurt performance, and if they are incompatible, we have performance guarantees for the higher-priority tasks, which is not the case for MORL methods that don't use priority.

4.

We have experiments that use more than two subtasks in Appendix G.2.

6.

$Q_i$ in Eq. (7) refers to the subtask Q-functions of an agent that is lexicographically optimal for all $n-1$ higher-priority tasks. These are not optimal Q-functions $Q_1^*, \dots, Q_n^*$. We have changed the sentence preceding Eq. (7) to emphasize this.

8.

As discussed in the introduction and related work section, the primary contribution of our paper is an algorithm for continuous action-space MDPs. A discrete version of PSQD would be conceptually similar to existing methods like value-based lexicographic MORL (Skalse et al. 2022), lexicographic value iteration (Wray et al. 2015), or TLO (Zhang et al, 2023), which all require finite action spaces.

9.

This is the standard lexicographic constraint consistent with related work. As we write in the paper $J_i$ are some performance measures (like value functions, Q-functions), the symbol $\pi$ is defined as policy, and the symbol $\varepsilon_i$ is defined, in the next line, a performance threshold scalar for subtask $i$.

We again thank reviewer SKG7 for their review of our work, which we believe allowed us to improve the understandability and reproducibility of our work, due to the added pseudocode and picographic overview.

We kindly thank reviewer tsYf for their review of our work. In the following, we answer the brought forward criticism and questions.

Reproducibility and reward functions

Reviewer tsYf is concerned that environments and reward functions are not sufficiently described.

Initially, we did not explicitly state the reward functions because, for reproducibility reasons, we plan to publish our complete codebase alongside the camera-ready version of the paper, which contains the reward functions and the complete environment specification. For completeness and to address this concern, we have now detail the reward functions in section G.1 in the appendix.

We hope this eliminates the concern w.r.t reproducibility.

Discussion on incompatible subtasks

Reviewer tsYf wishes for a more thorough discussion on situations where subtasks are incompatible and is concerned that our proposed method might fail in such cases since they believe that the indifference spaces would become empty.

We fully agree that this discussion is important, interesting, and benefits the understandability of our method. To address this point, we have added a new discussion section, based on the following points, to the revised manuscript in Appendix E.

• Firstly, when subtasks are semantically incompatible (e.g. when $r_2 = -r_1$, where $r_1$ rewards going to the left and $r_2$ rewards going to the right), it should be recognized that the MORL task is fundamentally ill-posed since no agent can behave optimally, at the same time, for two objectives that are entirely incompatible. This inherent challenge exists independently of the optimization method.
• Secondly, we want to highlight an advantage of our method over MORL methods without task priorities: If two objectives are semantically incompatible, our agent attempts to solve the lower-priority task as best as possible, while the constraint ensures that the higher-priority task is still solved optimally (up to the $\varepsilon$ threshold). If the lower-priority task rewards driving to the left, but the higher-priority task rewards driving to the right, our agent has to drive to the right, due to the lexicographic constraint. A non-lexicographic MORL agent will instead behave unpredictably in an attempt to maximize the incompatible subtasks at the same time (see Russel & Zimdars 2003 paper "Q-Decomposition for RL agents"). In fact, semantically incompatible subtasks were precisely the original motivation for lexicographic MORL (Multi-criteria Reinforcement Learning, 1998, Gábor, Kalmár and Szepesvári).
• Lastly, with our method, the indifference-space can never be empty (although very small). This is due to our problem definition (threshold relative to subtask Q-function) and the iterative nature of our learning algorithm. By first adapting the second-highest subtask solution using the lexicographic constraint of the highest-priority subtask, the second-highest priority subtask solution only assigns high value to actions that are in the highest priority subtask's indifference space. Thus, after adaptation, the indifference spaces of subtasks overlap, even when the original subtask rewards are semantically incompatible. Generally speaking, for some priority level $i$, in our adaptation step, all higher-priority subtask solutions are already "compatible" because they have already been adapted. Then, even in the extreme case where we set $\varepsilon_i = 0$, all lower-priority tasks can still "chose" from all optimal actions for task $r_i$. If, on the other, absolute performance thresholds were used that are not relative to the (adapted) subtask Q-functions, the intersection of those indifference spaces could indeed be empty.

Threshold ablation

Reviewer tsYf wonders about the sensitivity of our method to differing $\varepsilon_i$ thresholds and suggests an ablation study.

The behavior that our agent learns is indeed governed by the $\varepsilon_i$ thresholds since these thresholds give rise to the action indifference space for each subtask. Intuitively, larger $\varepsilon_i$ values mean that the performance for task $r_i$ is allowed to degrade more, in favor of increased performance for lower-priority tasks. Conversely, lower $\varepsilon_i$ values mean that the performance for task $r_i$ must remain more optimal, while lower-priority tasks are more restricted in their subtask optimization. We discuss in Section 6 of the paper how adequate values for $\varepsilon_i$ can be found practically.

To address this concern, we have conducted and added an ablation study on different $\varepsilon$ values in the appendix, Section H.3. This experiment empirically confirms the above-described relationship between $\varepsilon_i$ scalars and subtask performance.

We again thank reviewer tsYf for their review of our work, we firmly believe the review allowed us to add important and beneficial content to our paper.