We appreciate the thoughtful review and the constructive feedback. We hope the following clarifications and experiments address your questions.

Q1: Stronger baselines would strengthen the novelty argument.

Thanks for the kind suggestion. RCPO has been covered in our baseline PDQN-Rco. The Lagrangian dual method we adopt is conceptually similar to CVPO in addressing constrained problems; however, CVPO first computes a variational distribution and then solves the parameterized network based on this distribution, which introduces additional network estimation errors and is not well-suited for constrained hybrid-action policy optimization. We have discussed CVPO in our paper.

Q2: The theoretical bounds of the paper (e.g., $\Theta \left(\frac{\sqrt{|\mathcal{S}||\mathcal{A}|}}{(1-\gamma)^{1.5}\sqrt{K}}\right)$) depend on these idealized conditions, which may limit the practical scalability.

Thanks for the kind suggestion. Although our theoretical analysis is based on the assumption of accurate value estimation, we have already incorporated scaling in the analysis, as shown in the inequalities (i), (ii), (iii), and (v) in Eq. (16). Intermittent input noise affects training stability, while persistent noise slows down the convergence rate, but it does not have a significant impact on the final results.

Q3: How does CHPO perform when critic networks have high estimation errors (common in high-dimensional or noisy environments)?

Thank you for bringing our attention to the robustness. We add ablation tests with noisy cost and reward signals, where Gaussian noise is injected into the cost and reward signals to evaluate the robustness of CHPO when the critic networks exhibit higher estimation errors. The experimental results of Parking are presented below, showing the results under 1% and 5% persistent noise as well as random noise.

Based on the above experiments, we conclude that sporadic random noise does not affect the final results but causes fluctuations during training, while persistent noise slows down the convergence rate without impacting the final stable convergence. Moreover, in theoretical analysis, we apply scaling as shown in the inequalities (i), (ii), (iii), and (v) of Eq. (16). Based on both experimental and theoretical analysis, we believe that the approximation errors do not affect the final stable convergence.

Q4: Acronyms "C/A ratio" is introduced in Remark 4.5 but not defined until Appendix B.4.

Thanks for the kind suggestion. We have added the definition of the C/A ratio at line 246 of the manuscript, i.e.,“We define C/A ratio $= \frac{\sum_{i}^{m}|\mathcal{N}{c_i}|}{\sum_{i}^{m}|\mathcal{N}\_{c_i}| + |\mathcal{N}\_r|}$ as the proportion of update steps dedicated to cost minimization relative to the total number of policy updates.”

Q5: The paper states that CHPO is "inspired by HPPO [2]" but does not clearly contrast their architectures. It makes me a little bit confused to identify how CHPO differ from HPPO design.

Thanks for the comment. HPPO is a policy optimization method designed for hybrid action spaces, focusing primarily on reward maximization. However, it does not account for safety constraints, which often leads to cost violations when applied to safety-critical tasks.

CHPO builds upon the hybrid-action policy optimization framework of HPPO by introducing Constrained Parameterized-action Markov Decision Process (CPMDP), where safety constraints are explicitly modeled through a cost function $C$. CHPO proposes a constrained hybrid-action actor-critic architecture, consisting of two actors and two critics, where the two critics are responsible for learning the reward and cost state-value functions, respectively. During policy updates, CHPO accounts for safety constraints and rewards, employing a two-stage update mechanism of reward maximization and cost minimization, whereas HPPO optimizes solely for rewards. Although CHPO draws inspiration from HPPO in value function design, its core innovations lie in introducing CPMDP, the constrained hybrid-action actor-critic architecture, and constrained hybrid-action policy optimization, that are absent in HPPO design.

Q6: The paper does not explicitly analyze computational efficiency (e.g., training time, parameter count), which could be a concern for real-time applications.

Thank you for bringing our attention to the efficiency. The training time and parameter count of CHPO and the baselines are as follows:

Although CHPO has a longer runtime and a larger number of parameters compared to the baselines, their memory usage is similar, and CHPO achieves better performance.

Q7: The paper uses inconsistent formats and some spaces in the citation format have been ignored.

Thank you for your careful review. We have checked similar issues and revised the formatting of equation and figure references in the manuscript: “Eq. 5” has been changed to “Eq. (5)” at line 613, and “Fig.6” has been changed to “Fig. 6” at line 334. Additionally, we have added a period at the end of Eq. (20) between lines 599 and 600.

Q8: Remark 4.5 in the text uses a formatting style that does not align with the naming conventions of other structured elements (e.g., Lemma 4.2, Proposition 4.3).

Thank you for your careful review. We have adjusted the formatting style of Remark 4.5 in the text to match the naming conventions of Lemma 4.2 and Proposition 4.3.

Thank you for your valuable feedback. We appreciate the time and effort you have dedicated to reviewing our paper. We will address the weaknesses and questions you raised below.

Q1: The theoretical analysis does not appear to explicitly address or leverage PAMDP structure.

Thanks for the comment. Based on PAMDP, we divide the policy network into a discrete policy $\pi_{\theta_d}$ and a continuous policy $\pi_{\theta_c}$, which share the same backbone but differ in the output heads: one outputs discrete actions, while the other outputs continuous actions. For clarity, we denote their parameters as $\theta_d$ and $\theta_c$, respectively. Since $\theta_d$ and $\theta_c$ share the same optimization objective, we unify them as $\theta_p$. Our theoretical analysis is consistently conducted on the hybrid policy parameter $\theta_p$ derived from PAMDP. We define $|\mathcal{A}| := |\mathcal{A}_d| + |\mathcal{A}_c|$, which unifies the discrete and continuous action spaces in PAMDP into $|\mathcal{A}|$ for theoretical analysis. We have further clarified in the paper the relationship among $\theta_p$, $|\mathcal{A}|$, and PAMDP in theoretical analysis.

Q2: The evaluated environments lack diversity and are quite similar in nature.

Thanks for the kind suggestion. Currently, there is no existing simulation environment specifically designed for the constrained hybrid-action setting, so we have to modify and construct tasks for evaluation. We are applying our approach to real-world exploration environments, specifically to the RM65 robotic arm, where high-level motion primitives (fixed discrete actions) are manually designed, such as Lift (raising the end-effector upward) and Twist (rotating the gripper or object along a certain axis). The execution details of each discrete action are controlled by continuous parameters. We are in the process of applying CHPO to this robotic arm to accomplish hybrid-action tasks while ensuring safety.

Q3: What is the run-time/memory overhead of maintaining two critics and two actors compared to standard PAMDP methods?

Thank you for bringing our attention to the efficiency. We have recorded the run-time and memory usage of the standard PAMDP method (HPPO) and CHPO, as shown in the table below. Additionally, we have supplemented the run-time and memory usage of other baseline algorithms.

Q4: Evaluations are limited to four relatively low-dimensional tasks; performance on more complex robotics or vision-based environments remains untested.

Thank you for pointing this out. Our experiments are currently limited to low-dimensional tasks because there is no existing simulation environment for the hybrid-action settings. We modify and build tasks for evaluation. We are actively applying our approach to real-world RM65 robotic arm exploration. In this setting, the discrete actions of the robotic arm include operations such as lift, twist, and grasp, while the continuous actions correspond to the execution parameters for each discrete action. We perform constrained hybrid-action policy optimization on the robotic arm to accomplish tasks while ensuring safety.

Q5: Convergence bounds assume accurate value estimates and tabular |S|,|A|. However, since |A| includes continuous parameters, how is this handled in your analysis?

According to related work [1], the boundary of the critic network is related to the dimensions of $|\mathcal{S}|$ and $|\mathcal{A}|$, and the relationship holds in both discrete and continuous spaces $||Q_k - \hat{Q}_k ||_2 \leq \frac{\sqrt{(1-\gamma)|\mathcal{S}||\mathcal{A}|}}{\sqrt{K}}$. Building on [1][2], this work further defines $|\mathcal{A}| := |\mathcal{A}_d| + |\mathcal{A}_c|$, which remains applicable to the convergence analysis of our method. We have added clarifications regarding this in the revised manuscript.

Moreover, our work is based on tabular-form derivations, primarily focusing on analyzing and illustrating the decisive parameters affecting the convergence boundary of the CHPO algorithm. Therefore, we have not derived the convergence for network-based representations. If such a derivation is required, one directly replaces the parameterized representation with the weights and biases of the policy network, in which case the convergence boundary will incorporate the estimation error of the policy network. We have added this clarification to the revised manuscript.

Q6: How is the PAMDP structure incorporated into the analysis?

Thanks for the comment. Based on the PAMDP framework, the hybrid action space consists of a discrete action space $\mathcal{A}\_d$ and a continuous action space $\mathcal{A}\_c$. We define $|\mathcal{A}|:= |\mathcal{A}\_d| + |\mathcal{A}\_c|$ to represent this hybrid action space for subsequent theoretical analysis. Under the PAMDP framework, we employ a discrete policy $\pi_{\theta_d}$ to output discrete actions and a continuous policy $\pi_{\theta_c}$ to output continuous actions. In the implementation of the CHPO algorithm, these two policies share the same backbone network but are distinguished at the output heads to reduce computational overhead. Since the optimization objectives of $\pi_{\theta_d}$ and $\pi_{\theta_c}$ are identical, we unify them into the hybrid-action policy $\pi_{\theta_p}$ for the subsequent theoretical analysis, where $\theta_p$ is composed of $\theta_d$ and $\theta_c$.

Q7: What is the backbone of CHPO? Is it based on HPPO, PDQN, or PADDPG?

Thank you for catching this. CHPO is built upon HPPO. We chose HPPO because, among the three algorithms, HPPO demonstrates superior performance in handling purely hybrid-action tasks. PADDPG addresses hybrid-action tasks by converting the discrete action space into a continuous one, while PDQN separately applies DQN for discrete actions and DDPG for continuous actions. Compared to HPPO, both PDQN and PADDPG involve more complex handling of hybrid action spaces and show inferior performance. We also evaluate the performance of these three algorithms in terms of reward maximization without cost consideration. The results averaged over 3 seeds in Moving are as follows:

These results indicate that HPPO performs better on hybrid action tasks when safety constraints are not considered.

Q8: Why were HPPO-Rco, PADDPG-Rco, and PDQN-Lag not included for a more balanced comparison?

Thank you for catching this. PDQN cannot be combined with the Lag method to form PDQN-Lag because PDQN consists of two separate components: DQN for selecting discrete actions and DDPG for selecting continuous actions, making it infeasible to integrate the Lag method into both parts simultaneously. Therefore, we choose to combine PDQN with RCPO to form PDQN-Rco. In addition, we combine both HPPO and PADDPG with RCPO and conduct tests in the Parking environment. The comparative results under the cost limit $\bar{c}_i = 1$ are as follows:

Their performance is inferior to PDQN-Rco; therefore, we ultimately select PDQN-Rco as the baseline instead of HPPO-Rco and PADDPG-Rco.

Reference

[1] Dalal, G., Szorenyi, B., & Thoppe, G. (2020, April). A tale of two-timescale reinforcement learning with the tightest finite-time bound. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 3701-3708).

[2] Xu, T., Liang, Y., & Lan, G. (2021, July). Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning (pp. 11480-11491). PMLR.

Thank you for your response. Regarding the theoretical analysis, the convergence bounds of RL policies for both continuous and discrete actions are positively correlated with the dimensionality of the observation and action spaces. Although our work is based on the hybrid action PAMDP framework, both value estimation and policy updates are conducted following the principles of proximal policy gradients. Therefore, the theoretical derivation for the hybrid action setting in our work still fundamentally relies on theoretical foundations developed for single action space RL. As a result, apart from the dimensional definition of the hybrid action space, the conclusions we derive remain consistent with the theoretical foundations established in prior work [1].

Reference

[1] Dalal, G., Szorenyi, B., & Thoppe, G. (2020, April). A tale of two-timescale reinforcement learning with the tightest finite-time bound. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 3701-3708).

Thank you for your further feedback. We’re grateful for the opportunity to improve our work. We are currently integrating CHPO into the DLPA framework and will provide the corresponding experimental results as soon as possible. If time does not permit, we will include the results in the appendix of the revised manuscript.

We sincerely thank the reviewer for their insightful feedback, and we’re grateful for the opportunity to improve and clarify our work through answering your detailed questions and concerns.

Q1: The algorithm requires two actor/critic networks, which doubles the computational burden and makes it difficult to apply to large neural networks, i.e., LLMs.

Thank you for catching this. The addition of the cost value estimation network indeed increases some computational overhead. However, in our study, the policy network is not composed of two independent networks. Instead, the discrete and continuous action policies share a common backbone network, with two separate output heads generating the discrete actions and the continuous parameters, respectively.

Our work primarily focuses on the constrained policy optimization problem for agents that take structured parameters as input and output actions in hybrid action spaces. If this approach is extended to large neural networks, the constrained policy optimization of our algorithm is still applicable. By leveraging methods such as GRPO [1] and replacing value estimation networks with statistical sampling, the critic networks are removed.

Q2: This paper does not demonstrate any algorithmic justification for using a “hybrid-action” setup.

Thanks for the comment. Hybrid-action RL algorithms are designed to handle special scenarios where discrete actions and their associated continuous parameters are selected simultaneously. Conventional RL is limited to either purely discrete or continuous action spaces and cannot address hybrid-action problems. In real-world scenarios, such as a soccer game, the agent selects a discrete action (e.g., dash, turn, tackle, or kick) and determines continuous actions such as power and direction to successfully perform a shooting task. Conventional RL methods do not simultaneously handle the selection of discrete actions and the determination of continuous actions. In contrast, hybrid-action RL policy outputs both the selected discrete action and its corresponding continuous parameters simultaneously.

The parameters $\theta_d$ and $\theta_c$ are not treated as a single continuous policy. Unlike PADDPG, which converts discrete actions into continuous action spaces, our policy network shares a single backbone network with two distinct output heads: one outputs the discrete actions and the other outputs the continuous actions. The differences between the two actor networks arise solely from their separate output heads, and we denote their parameters as $\theta_d$ and $\theta_c$ to distinguish them. Since the optimization objectives of $\theta_d$ and $\theta_c$ are aligned, we unify them under the parameter $\theta_p$ to jointly update both discrete and continuous policies.

In our constrained hybrid-action setting, the agent not only selects both the discrete action and the continuous action simultaneously but also satisfies safety constraints. For example, in the Parking task, the agent needs to determine the curve type and its arc length while avoiding collisions with surrounding obstacles.

Q3: The authors should compare to existing contained PPO algorithm [2] and discuss the difference to it.

Thanks for the kind suggestion. Compared with the existing contained PPO algorithm, our approach handles constrained policy optimization in hybrid action spaces. The existing contained PPO algorithm focuses on maximizing rewards under safety constraints within a single action space and does not account for hybrid-action scenarios. In contrast, our method achieves constrained policy optimization in discrete–continuous hybrid action spaces. To clearly highlight the differences from [2], we have revised the related work section of the manuscript to explicitly discuss that the “existing constrained PPO algorithm [2] focuses on constrained policy optimization in a single action space and does not consider hybrid-action scenarios.”

Q4: What the connection is between the proposed algorithm and the hybrid-action setting in this paper?

Building on the foundation of constrained hybrid-action spaces, our work further addresses the critical issue of safety. CHPO integrates cost constraints into the hybrid-action policy optimization process, ensuring that the policy not only achieves high rewards but also satisfies safety constraints when jointly optimizing discrete actions and continuous actions. In our hybrid-action setting, the agent simultaneously considers both the selection of discrete actions and the determination of continuous actions while ensuring safety. As illustrated in Fig. 1 of the manuscript, conventional hybrid-action RL methods cannot learn safe policies in discrete-continuous hybrid action spaces. Our proposed algorithm specifically targets the problem of constrained policy optimization in the hybrid-action setting.

Reference

[1] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., ... & Guo, D. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

[2] Jayant, A. K., & Bhatnagar, S. (2022). Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems, 35, 24432-24445.

Additionally, we conducted a Parking experiment using the conventional RL method by simplifying the hybrid action space into a single continuous action space without considering safety constraints, and compared the results against those obtained using the hybrid-action RL algorithm.

A bit weird here. I think it would only be fair to compare RL methods that simplify the hybrid action space into a single continuous action space then optimize with safety constraints. Why is the comparison done without constraints here?

Thanks for the responses! Some of my concerns have been addressed, but I still have a question.

Conventional RL is limited to either purely discrete or continuous action spaces and cannot address hybrid-action problems.

Why? discrete space is a specific case of continuous space. In my understanding, these two spaces can be directly simplified into a single continuous space. I can understand the hybrid setting considered in this paper; I just don't understand why we can't first simply the two spaces into a single continuous one and then directly apply existing constrained PPO algorithm [1].

Thank you for your further response. We hope the following explanation addresses your concerns.

Q1: Given that the main difference between this paper's algorithm and [1] lies in the hybrid action space setting, the authors should compare against the algorithm in [1] to demonstrate the negative effect of relaxing the space, rather than comparing to unrelated prior methods.

Thanks for the comment. The work [1] implements constrained policy optimization using Lagrangian multipliers. To apply it to the hybrid action space setting, the discrete action space within the hybrid action space needs to be converted into a continuous action space. This is the same processing idea used by the earlier method PADDPG [2], and PADDPG has already been shown to remain effective under such a transformation. In our study, we also employ Lagrangian multipliers to perform constrained policy optimization after relaxing the hybrid action space into a continuous action space. In essence, PADDPG-Lag is equivalent to work [1], and PADDPG has already been demonstrated to adapt well to hybrid action spaces. Moreover, the network architecture and negative log-likelihood loss used in the model-learning component of the work [1] are not specifically designed for this “relaxation–projection” scenario.

The core of the CHPO algorithm lies in: (i) addressing policy outputs in a hybrid action space to handle discrete–continuous action selection, and (ii) solving the constrained optimization problem using primal policy optimization without introducing additional hyperparameters. The work [1] proposes a model-based safe RL algorithm that leverages a dual method to address safety constraints and achieves good performance. However, there are significant differences between their approach and ours, including: (1) the type of action space being solved, (2) the method used to handle safety constraints, and (3) whether an additional environment model is required. We have discussed these key differences between our algorithm and that of the work [1] in detail in both the introduction and related work sections of the manuscript.

Q2: By the way, apart from the hybrid setting, are there any other differences between the algorithm in this paper and the one in [1]? Intuitively, the theoretical part seems essentially the same. Considering that the hybrid setting doesn’t seem to introduce additional challenges for the theoretical analysis, am I misunderstanding something?

Thank you for pointing this out. The work [1] first integrates real-world data with a learned model to perform online learning of environment dynamics, and then applies PPO-Lagrangian on truncated predicted trajectories to update both the policy and the Lagrangian multiplier, thereby achieving policy optimization under safety constraints. While the work [1] uses a Lagrangian method for constrained policy optimization, our approach employs primal policy optimization to address the constrained policy optimization problem. Moreover, the work [1] does not provide a detailed theoretical analysis. There are notable differences between work [1] and our algorithm, including: (1) the type of action space being addressed; (2) the method used to handle safety constraints; and (3) whether an additional environment model is required.

In our study, the hybrid-action outputs satisfy the convergence properties of both the discrete action and continuous action value estimations as well as the policy networks. Therefore, building on the theoretical foundations established in prior research, we extend the action space and derive the convergence properties of our algorithm. Our theoretical analysis primarily focuses on the convergence characteristics of the safety constrained problem after the action space expansion, and the derivations in our work are centered on this aspect in detail.

Reference

[1] Jayant, A. K., & Bhatnagar, S. (2022). Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems, 35, 24432-24445.

[2] Hausknecht, M., & Stone, P. (2015). Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143.

We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.

Q1: The probability of risk occurrence at the initial state cannot be avoided.

Thank you for bringing our attention to the offline setting. You raise a highly valuable and thought-provoking question. Safety constraints in hybrid action spaces are also in high demand in real-world applications, such as in gaming[1][2] and in recent IL + Online RL studies[3][4]. Our work simultaneously addresses the problem of safe policy optimization in hybrid action spaces, providing both a theoretical foundation and technical support for future research on constrained hybrid-action policy optimization in offline settings. Moreover, real-world data is used to build high-fidelity scenarios through world models, providing a feasible paradigm for online closed-loop interactions in safe RL. This makes it possible to sample from real-world environments with high safety requirements.

The relatively high cost of CHPO in a long iteration is intentional to ensure global optimality. We incorporate the C/A ratio during training to ensure that the cost decreases gradually, allowing the algorithm to explore and achieve the maximum reward while satisfying the safety constraints. Furthermore, if higher policy safety is desired, we adjust the hyperparameters to balance the trade-off between reward and safety constraints, thereby ensuring strict adherence to safety constraints.

Q2: The variance may be too large.

Thanks for the kind suggestion. Although the theory guarantees the safety of the policy, in practical implementations the bias in the value estimation of the cost value network occasionally cause the policy to exceed the cost threshold. However, if strict adherence to the cost constraint is required, our algorithm achieves this by tuning the hyperparameters to ensure that the final policy strictly satisfies the cost constraint. Furthermore, to enhance the rigor of our statements, we have adopted your suggestion and revised the manuscript to clarify that “CHPO maximizes the average reward while ensuring that the average cost across multiple seeds satisfies the safety constraint.”

Q3: The baselines are out of date.

Thank you for your thoughtful review. The baselines we use are classical approaches, which have also been referenced by other recent studies [5][6]. Since there is no existing algorithm specifically for constrained hybrid-action RL, we combine well-established hybrid-action RL methods with stable constrained RL algorithms.

Q4: The computation of the reward advantage function in Line 190 can be described in the appendix.

We appreciate your careful reading. We provide the computation of the reward advantage function in the appendix: $\hat{A}^{r}_t=-V^{r}(s_t)+r_t+\gamma r\_{t+1}+\cdots+{\gamma}^{T-t-1}r\_{T-1}+{\gamma}^{T-t}V^{r}(s_T)$.

Reference

[1] Hausknecht, M., & Stone, P. (2015). Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143.

[2] Fan, Z., Su, R., Zhang, W., & Yu, Y. (2019). Hybrid actor-critic reinforcement learning in parameterized action space. arXiv preprint arXiv:1903.01344.

[3] Gao, D., Wang, H., Zhou, H., Ammar, N., Mishra, S., Moradipari, A., ... & Zhang, J. (2025). IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning. arXiv preprint arXiv:2505.10442.

[4] Albaba, M., Christen, S., Langarek, T., Gebhardt, C., Hilliges, O., & Black, M. J. (2024). Rile: Reinforced imitation learning. arXiv preprint arXiv:2406.08472.

[5] Niu, Y., Xu, J., Pu, Y., Nie, Y., Zhang, J., Hu, S., ... & Liu, Y. (2021). Di-engine: A universal ai system/engine for decision intelligence.

[6] Zhang, R., Fu, H., Miao, Y., & Konidaris, G. (2024). Model-based reinforcement learning for parameterized action spaces. arXiv preprint arXiv:2404.03037.