We sincerely thank you for your valuable feedback on our manuscript, and for highlighting the broad applicability of our proposed action masking approach. We would like to address your concerns and questions in the following.

Weaknesses

Action masking criteria

Thank you for sharing your assessment. There can be different notions of relevance, which has implications for required task knowledge. A relevant action set could encode a formal specification, guarantee the stability/safety of a dynamic system, or enforce system constraints involving interdependent actions (see summary rebuttal A2 for more details). For an explicit relevant action set, one can determine if it is convex. If so, our continuous action masking can be applied. If not, the relevant action set could be underapproximated with a convex set (see rebuttal for reviewer PubC, Q1 for more details). We will clarify the notion of relevance in the introduction and extend Sec. 4.3 with more details on application limitations originating from the assumptions.

Unconvincing experiments

We appreciate your suggestion to include a different type of task with a more intuitive relevant action set. We will extend our empirical evaluation with the Walker2D MuJoCo environment. Here, the Walker2D is interpreted as a battery-driven robot and the relevant action set is based on the total power constraint of the battery. Such a constraint encodes the interdependence of multiple action dimensions and cannot easily be enforced with vanilla RL. Please refer to the summary rebuttal for more details on the new experiment (see A3) and more examples for relevant action sets (see A2).

Questions

Related work as separate section

Thank you for this remark. We agree and will move the related work subsection to a dedicated section.

Clarification on the effect of G being square and non-singular

Thank you for this question. A generator matrix $G$ that is square and non-singular does restrict the possibilities for relevant action sets, e.g., in two dimensions only rectangles and parallelotopes can be represented but not hexagons. We state in the paper that for this special case of an invertible matrix $G$, the policy gradient for the generator mask does not change (see Proposition 5). You correctly note that it is likely that the generator dimensions $P$ are larger than the action dimensions $N$. Yet, there are cases where relevant action sets with $N=P$ are a valid choice. For example, if the relevant action sets are state-dependent interval sets as for the 3D Quadrotor or if there is a linear dependency between the action dimensions. The latter is the case for the 2D Quadrotor where the force at the left and right rotors should be similar to avoid flipping over. Thus, a parallelotope shape would be a valid choice. Note that existing continuous action masking on interval sets would only poorly cover parallelotope relevant action sets as visualized in the rebuttal PDF, Fig. 2. We will reflect this answer in the paper by clarifying the implications of different choices of $G$ in Sec. 3.2.

Limitations

Action masking for value-based RL

Thank you for your comment. To the best of our knowledge, there is no popular deep RL algorithm for Q-learning in continuous action spaces that is not an actor-critic. For discrete action spaces, previous action masking work has demonstrated its effectiveness [4,6,11,22] and continuous relevant action sets could be discretized to make these approaches applicable. In our limitations, we comment on TD3 and SAC, which both learn a policy and for which future work is required to determine the theoretical and empirical effects of action masking on the policy. Nevertheless, we see that our statement in Line 149 could be misleading:

Since the size of the output layer of the policy or Q-function network cannot change during the learning process

We will change it to:

Since the size of the output layer of the policy network cannot change during the learning process

Obtaining relevant action sets and relevance of action masking

Thanks for your comment. We see that our paper lacked a more explicit specification of the notion of action relevance and a clarification of the practical relevance and benefits of the formulation as part of the policy. We will add this in the revised paper and would like to refer you to A1. and A2. of the summary rebuttal for a detailed answer.

References:

[4] Feng et al. 2023

[6] Fulton et al. 2018

[11] Huang et al. 2022

[22] Rudolf et al. 2022

We sincerely thank you for your valuable feedback on our manuscript. We are particularly grateful for the acknowledgment of our manuscript's originality and significance. In the following, we outline how we incorporated your feedback and clarify open questions.

Improvements to the mathematical formulation

Thanks for pointing out notation improvements. We reworked the preliminaries and methodology sections to improve notational precision. Specifically, we corrected the following points:

• $r \rightarrow: \mathcal{S} \times \mathcal{A}$ to $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ in [L84].
• in the ray mask approach: $g(a) = c + \frac{\lambda_{\mathcal{A}}(a)}{\lambda_{\mathcal{A}^r}(a)} (a - c)$ to $g(a) = c + \frac{\lambda_{\mathcal{A}^r}(a)}{\lambda_{\mathcal{A}}(a)} (a - c)$ in [Eq. 6, L126]
• in the generator approach
  • $g : \mathcal{A} \rightarrow \mathcal{A}^r$ to $g : \mathcal{A}^l \rightarrow \mathcal{A}^r$ in [L163]
  • $G \in \mathbb{R}^{P \times N}$ to $G \in \mathbb{R}^{N \times P}$ in [L154].
  • Now we have $G \in \mathbb{R}^{N \times P}$, so the dimensions in [Eq. 12, L164] fit correctly.

Clarification of the relevant action space

Thank you for highlighting that the notion of action relevance was not properly defined. We will added a formal definition of the relevant action set to the preliminaries:

We further assume that we can compute a state-dependent relevant action set $\mathcal{A}^r(s) \subseteq \mathcal{A}$, which potentially reduces the action space, based on task knowledge.

To introduce our notion of relevant actions, we are adding to the first paragraph of the introduction

Irrelevant actions are actions that are either physically impossible, forbidden due to some formal specification, or evidently counterproductive for solving the task.

Additionally, we will clarify that only the relevant action set $\mathcal{A}^r(s)$ is state-dependent, the action sets $\mathcal{A}$ and $\mathcal{A}^l$ are not. We would like to refer the reviewer to the summary rebuttal A.2 Computing relevant action sets for a more in-depth discussion on this topic.

Proof of bijectivity of $g(a)$ in the ray mask

Lemma: The mapping function $g: \mathcal{A} \rightarrow \mathcal{A}^r$ of the ray mask $g(a) = c + \frac{\lambda_{\mathcal{A}^r}(a)}{\lambda_{\mathcal{A}}(a)} (a - c)$ is bijective.

Proof: We prove that $g(a)$ is bijective by showing that it is both injective and surjective.

For any convex set $\mathcal{A}^r$ with center $c$, we can construct a ray from $c$ through any $a^r \in \mathcal{A}^r$ as $r_d(t) = c + d t$, where $d = \frac{a-c}{\Vert a-c \Vert_2}$, and $t \in \left[ 0, t_{max} \right]$ limits the ray to $\mathcal{A}$. Since $\mathcal{A}^r \subseteq \mathcal{A}$, this also holds $\forall a \in \mathcal{A}$. By construction, any two distinct rays only intersect in $c$. For all points on a given ray, the scaling factors $\lambda_\mathcal{A}$ and $\lambda_{\mathcal{A}^r}$ are constants, allowing us to rewrite $g(a)$ for all $a=r_d(t)$ as the linear function:

$$g \left( r_d(t) \right) = c + \frac{\lambda_{\mathcal{A}^r}}{\lambda_{\mathcal{A}}} \left( r_d(t) - c \right).$$

To show that $g(a)$ is injective, i.e. $\forall a_1, a_2 \in \mathcal{A}$, if $a_1 \neq a_2 \implies g(a_1) \neq g(a_2)$, consider the following two cases for $a_1 \neq a_2$.

Case 1: If $a_1$ and $a_2$ are on the same ray, then, $a_1 = r_d(t_1)$ and $a_2 = r_d(t_2)$ with $t_1 \neq t_2$. Since $g \left( r_d(t) \right)$ is linear and thereby monotonic, it follows that $g \left( r_d(t_1) \right) \neq g \left( r_d(t_2) \right)$ and consequently $g(a_1) \neq g(a_2)$.

Case 2: Otherwise, $a_1$ and $a_2$ are on different rays and $a_1 \neq a_2 \neq c$, so $g(a_1) \neq g(a_2)$ follows directly as the rays only intersect in $c$.

Thus, $g(a)$ is injective.

For showing $g(a)$ to be surjective, it is sufficient to show that $\forall a^r \in \mathcal{A}^r$, there exists an $a \in \mathcal{A}$, for which $g(a) = a^r$.

Consider the ray $r_d(t)$ passing through $a^r$. We know that $g \left( r_d(0) \right) = g(c) = c$, and $g \left( r_d(t_{max}) \right)$ is the boundary point of $\mathcal{A}^r$ from $c$ in the direction $d$. Moreover, $a^r$ lies on the line segment between $c$ and $g \left( r_d(t_{\max}) \right)$. Since $g \left( r_d(t) \right)$ is linear and continuous, according to the intermediate value theorem, $\exists t^* \in \left[ 0, t_{\max} \right]$, for which $g \left( r_d(t^*) \right) = a^r$ for any $a^r \in \mathcal{A}^r$ along the ray $r_d(t)$. As we can construct such a ray through any point in $\mathcal{A}^r$, we have shown that for every $a^r \in \mathcal{A}^r$, there exists an $a \in \mathcal{A}$ such that $g(a) = a^r$, thus proving surjectivity.

Additional experiments required

Based on your recommendations, we added further evaluations as discussed in A3. of the summary rebuttal. Additionally, we will add an illustration of typical rollouts on the Seeker environment as interpretable qualitative result (see rebuttal PDF, Fig. 4).

Rationale behind the design choices for action relevance in our environments

Thank you for your question regarding the design choices of our relevant action space. A key advantage of our masking approaches is that they ensure that only relevant actions are executed and therefore can be used for safety-critical tasks. There are other notions of relevance, which we explore in the summary rebuttal A2. Here, we choose experiments where the relevant action set is a safe action set. In particular, a relevant action is one that avoids collision with an unsafe region. For the Seeker environment, the unsafe region is defined by an obstacle (see Fig. 2). For the quadrotor stabilization experiments, we compute a control-invariant set, which acts as relevant state set $\mathcal{S}^r$.

We will clarify in 4.1.1 and 4.1.2 the motivation and computation of the relevant action set.

Thanks for your question. Fig. 4 shows qualitative deployment behavior for the different RL agents in the Seeker environment. We use the same goal state (gray circle), same obstacle (red circle), and same ten initial states (colored crosses) for the agents for better comparability. The ten trajectories have the same color as their initial state. Note that for the masking approaches the trajectories are almost identical after passing the obstacle and therefore plotted on top of each other.

The masking agents (ray, generator and distributional) reach the goal state for all ten initial states, with the generator being the quickest. In contrast, the PPO baseline agent reaches the goal with four out of the ten initial states and otherwise collides with the obstacle. The qualitative behavior is aligned with the quantitative results in Table 1 of the submitted paper meaning that the generator mask agent performs best and the PPO baseline performs worst with respect to reaching the goal efficiently.

The goal of the visualization in Fig. 4 is to provide an intuition on the trained agents' behavior. We choose the Seeker environment since we believe it is the most intutive to understand and thus also to interpret the qualitative behavior.

Note that there is unfortunately a typo in the subplot titles: distribution should be distributional.

Thank you for your insightful comments and for recognizing the novelty and theoretical grounding of our proposed methods. In the following, we respond to the weaknesses (W1 and W2) stated and questions (Q1 - Q6) raised.

Weaknesses

W1. Distributional mask being off-policy by nature

Thank you for sharing your reading of our distributional mask approach. Could you reiterate on why you would refer to the distributional mask approach as off-policy? We would refer to it as on-policy as we are directly sampling from the relevant policy $\pi_\theta^r(a^r|s)$, which is normalized through the integral $\int_{\mathcal{A}^r} \pi_\theta(a | s) \mathrm{d} a$, and derive the corresponding gradient $\nabla_\theta \log \pi_\theta^r(a^r | s)$. However, our implementation approximates the gradient by treating the integral as a constant, which introduces some off-policyness. However, such off-policyness is also introduced by most stochastic on-policy RL implementations (including PPO) through the clipping of samples from the normal distribution.

W2. Simplicity of experiments

Thank you for your assessment. The main goal of our experiments is to provide initial empirical evidence for continuous action masking and compare the three masking approaches. To strengthen our evaluation, we will add a Walker2D Mujoco environment and an additional baseline to the revised paper as described in A3. of the summary rebuttal and 3. in rebuttal for reviewer 8567.

Questions

Q1. Convex set assumption

Thanks for your question. Note that convex sets are an important generalization of current practice that allows for interval sets only. We believe that assuming relevant action sets are convex is reasonable since exclusion of extreme actions and dependencies between action dimensions can be described by them. Additionally, one can underapproximate a non-convex relevant set with a convex relevant set at the cost of excluding some relevant actions [https://arxiv.org/abs/1909.01778]. For disjoint convex sets, our continuous action masking could be extended with a discrete action that represents a decision variable switching between the disjoint subsets. To find an optimal policy, hybrid RL algorithms could be used [https://arxiv.org/abs/2001.00449]. Yet, this major extension of continuous action masking is subject to future work, and we will add these considerations in Sec. 4.3.

Q2. Mapping as part of the environment

Thank you for your question. Formulating action masking as part of the policy leads to more intuitive action spaces, potential integration of the masking map in the gradient, and more possible masking approaches (see summary rebuttal, A1). Note that implementing the ray mask on the policy or environment side is mathematically identical due to the unchanged gradient. To assess the empirical relevance of our formulation, we added another baseline where as part of the environment dynamics irrelevant actions are replaced with relevant ones. Please refer to 3. in rebuttal for reviewer 8567 for more details.

Q3. Gradient of the distributional mask

You are correct in your assessment that $\pi_\theta(a | s)$ is not an accurate probability of sampling action $a \in \mathcal{A}^r$ at state $s$. To address this issue, we normalize the constraint distribution with the integral in equation (15) and use the probability of the resulting distribution $\pi_\theta^r(a | s)$ for sampling and in the objective function. With this formulation, the entire probability mass of the resulting distribution $\pi_\theta^r(a | s)$ lies within $\mathcal{A}^r$, and the distribution accurately reflects the actual likelihood of sampling action $a^r$.

Q4. Goal color

Thanks for the hint. We will change the color in the revised paper.

Q5. Dynamics to the appendix

Thanks for the suggestion. We will extend Sec. 4 with the additional experiments as described in our summary rebuttal and move the details of the dynamics to the appendix.

Q6. Experiment with higher action dimensions

Thank you for your suggestion to extend our evaluation. To provide empirical insights for higher dimensional action spaces, we will add the Walker2D Mujoco environment as described in the summary rebuttal (see A3). Although the Walker2D has less than ten action dimensions, it has the highest action dimensionality of the four environments with six dimensions.

The compute costs are mainly affected by the calculation of the relevant action set in each state and by the additional operations for the mapping function $\mathcal{F}$ of the specific masking approach. The former depends strongly on the notion of relevance. For our experiments with collision avoidance as notion of relevance, we compute the relevant action sets at each state with an exponential cone program, which scales polynomially with system dimensions (i.e., state and action) assuming a suitable interior-point solver [Boyd, 04, Convex Optimization]. The second aspect important for the compute costs is the specific masking approach. For the ray mask, the cost is dominated by computing the boundary points, which is a linear program for zonotopes with polynomial complexity in the state dimension [Kulmburg, 21, On the co-NP-Completeness of the Zonotope Containment Problem]. For the generator mask, the matrix multiplication of $G$ with $a^l$ is the dominating operation. For the distributional mask, sampling with the random-direction hit-and-run algorithm introduces computational cost. With zonotopes, each step involves solving two linear programs to determine the boundary points. As mentioned in line 190 the mixing time, or the number of steps before accepting a sample, is set to $N^3$, where $N$ is the action dimension.

We will clarify the computational cost scaling in Sec. 4.3 and extend our runtime table in Appendix A.6 with the Walker2D environment, for which the PPO baseline and the generator mask training run at the same speed.

We thank you for your thoughtful and critical comments which helped us to strengthen our arguments for the utility of action masking. We address your questions below.

1. General applicability of this paper's ideas

Thank you for pointing this out. We agree that the two statements appear contradictory at first glance. However, "can" removes the contradiction in our view. In line 5, “can be” indicates that in some cases, minimal task knowledge might be enough to identify relevant actions. Using "can be obtained" in lines 284-285 highlights that while it is possible to obtain a relevant action set, it may also be quite challenging in many practical situations. Thus, our seemingly contradictory statements highlight the spectrum of difficulties to obtaining a relevant action set, on which we elaborate further in the summary rebuttal (A2). We will highlight this spectrum better in the revised version of this paper.

As we mention in the paper, action masking can be especially useful for safety-critical applications, where relevance means collision-avoidance. Regarding your first point, we provided such concrete examples for the application of autonomous driving in A2 of the summary rebuttal. To address the second point, we conducted an additional experiment for the MuJoCo Walker2D task, which we justify and evaluate in A3 of the summary rebuttal.

2. Gains not coming from the policy gradient, but only because of constraining the action space

Thanks for your question. We defined action masking as transforming the unmasked policy $a \sim \pi_\theta(a | s)$ to the masked policy $a^r \sim \pi_\theta^r(a^r | s)$, for which $a^r \in \mathcal{A}^r$ always holds. While most stochastic policy gradient algorithms utilize normal distributions with well-known policy gradients $\nabla_\theta \log \pi$, masked policies generally do not follow standard normal distributions. Consequently, we derived gradients for these masked distributions to provide a mathematically sound method. It's important to note that we do not claim these adapted gradients necessarily improve algorithm performance in general.

Nevertheless, it is not true that all masked policy gradients always reduce to the unmasked policy gradient $\nabla_\theta \log \pi_\theta(a | s)$. This only applies to the ray approach, and the proof for it is not trivial, making it also an important contribution. For the generator mask, the policy gradient remains the same, only if the generator matrix is invertible (line 173), which occurs solely when the zonotope has $2^N$ vertices. The distributional mask yields a substantially different gradient (see equation (17)). In practice, we approximate it with the original gradient, since the integral over $\mathcal{A}^r$ is intractable. We acknowledge this simplification in Sec. 4.3, and suggest potential approximations for future work.

Our primary intention was not to claim that the derived gradients universally improve RL algorithm performance. Rather, we emphasize that deriving correct gradients for modified distributions is crucial for establishing a mathematically sound method, thus forming an essential part of our contribution.

Regarding your second observation, we agree that most of the learning of vanilla PPO goes into the effort of learning to select relevant actions. However, we see this as a justification for the utility of action masking because it allows us to encode task knowledge directly into the policy and, thereby reducing the exploration space. Nevertheless, we do not intend to claim that action masking is inherently better than standard PPO, which we will make sure to clarify in our revised paper. Yet, our experiments do show that utilizing relevant action sets with action masking can drastically improve the convergence speed, albeit at a higher computational cost.

3. Simpler baselines for continuous action masking

Thank you for proposing two additional baselines for comparison. However, we do think that our implementation and experiments already cover both of them.

The ray mask is a version of the first proposed baseline. As derived for proposition 1, the ray mask does not affect the gradient of the policy distribution (because $g(a)$ is bijective), which makes it mathematically equivalent to being defined as part of the environment. We further address the proposal of generally viewing action masking as part of the environment in A1 of the summary rebuttal.

The second proposed baseline is mathematically equivalent to our distributional mask, since it only modifies the sampling procedure to use rejection sampling instead of geometric random walks. We initially used rejection sampling, but quickly found that it carries a significant downside, which makes it not applicable to RL. Consider the case in two dimensions, where $\mathcal{A}^r$ is a small set centered at $\left[ -0.5, -0.5 \right]^T$. If the policy $\pi_\theta(a | s)$ defines a normal distribution with mean at $\left[ 1.0, 1.0 \right]^T$ and small variance, the likelihood of a sample $a \sim \pi_\theta(a | s)$ being inside $\mathcal{A}^r$ is almost zero. This can cause the algorithm to get stuck at a sampling step, which we observed in practice. The issue even aggravates in higher dimensions due to the curse of dimensionality.

Nevertheless, we acknowledge that the PPO baseline may be considered relatively weak. We add a comparison to another common approach for adapting the actions of RL agents; action replacement [14]. This method substitutes actions outside the relevant action set with randomly sampled actions from within it. We depict the comparison in Fig. 1 of the rebuttal PDF. In the experiment, masking performs as good as action replacement in the 2D Quadrotor task, better in the 3D Quadrotor, and significantly better in the Seeker environment. These result suggests that action masking is a competitive approach with respect to a baseline that also explores the relevant action set only.

[14] Krasowski et al. 2023

Thank you for your rebuttal response. I am posting here for a potentially joint discussion with other reviewers (like PubC) who shared my question about treating action masking as part of the environment.

I want to understand this part better. I still believe the main benefit of formulating action masking as part of the policy comes from how it can help the policy parameters learn about what the action mask is, and thus, help it learn a better policy. For this to happen, there has to be a change in the gradient objective. I have been thinking about this in depth, and have some thoughts / questions.

1. Let's think intuitively at a high level. Is it even physically possible that gradient information can be passed into the policy? Say, I am a neural network (policy) and my outputs are transformed in a particular way, via $\mathcal{F}$ from $\pi(a|s) \to \pi^r(a^r | s)$, and then it goes through a non-differentiable operation (environment, E) to give a return R. Now, the gradients for me can still only flow through my outputs, i.e., $\pi(a|s)$. Why would separating what happens after my output be any helpful to me, i.e., how can I extract some information by separately treating $\mathcal{F}$ and E? Even if I knew what $\mathcal{F}$ is, I don't know what E is, and I can only see the final effect in the form of the return R. Is the knowledge of $\mathcal{F}$ adding any extra information above the knowledge of return R, or is my gradient conditionally independent of $\mathcal{F}$ given R?

2. For the case when $a^r = g(a)$ and g is bijective, we know that the gradient of $\pi^r$ = gradient of $\pi$. Then, why do we believe that when g is not bijective there will be any extra information added into my gradient?

It would be ideal if we could see some actual effect of gradient flow into the policy due to the knowledge of the action mask in any one of the methods and experimental results. That would resolve all my concerns. But even if that's not tractable right now, I would like to see some evidence or a thought experiment to prove that it is indeed possible!

3. I also tried to understand what the distributional mask's gradient in Eq. 17 physically implies. If we look at the terms in the gradient, the first term is the standard REINFORCE term, which has the implication that if the associated return R is high with this term, then the policy gradient increases $\log \pi_\theta(a | s)$ for that state. Similarly, it would reduce the second term here because of the negative sign, i.e., $\log \int_{A^r} \pi_\theta(a|s)da$ should go down if the return is high. But, this means that the probability density of $\pi_\theta(\cdot|s)$ should go down over $A^r$. Basically, the area under curve for the segment of $A^r$ has a lower probability density than the other areas in the action space. But, I don't see any reason why this should happen? This would increase the probability of "invalid actions" (along with the action $a$ from the first RL term). I see no reason why that would be helpful at all or why that makes sense?

To sum up, I really want to believe that there is some performance benefit to considering action masking as part of policy and not environment. But, I don't have any evidence to support that yet. I hope the authors can help me understand this.

If it turns out that there is no gradient flow, then I don't see any merit in the entire mathematical formulation of this paper, because all it says is that we should just consider action masking as part of environment, which is the trivial thing to do. I sincerely hope that is not the case, and by answering 1,2,3, we can get to that conclusion.