Model-based Reinforcement Learning for Parameterized Action Spaces

Thank you for the insightful feedback on our submission. We believe we can address your concerns. We have fixed the grammar errors accordingly in our updated manuscript.

Q1: “The author's description of the key innovations in this paper, such as the calculation of the H-step loss in model learning or the introduction of the CEM method, does not place enough emphasis on what makes these approaches unique when applied to the specific problem scenario PAMDP.”

A1: We believe the primary contribution of this paper is the very first model-based RL algorithm for the PAMDP setting. It outperforms all the previous model-free RL methods - many of which have been published at ICLR, including one just last year - for PAMDP on 8 standard PAMDP benchmarks. The PAMDP setting is very common in real-world tasks, and has drawn a lot of attention in recent papers, but all of them are model-free methods. We hope our paper can draw other researchers' attention to the model-based aspect of this problem setting since DLPA performs better than even the most sophisticated model-free methods. Calculation of the H-step loss in model learning is an important technique that we found to be especially useful in practice. Additionally, for the CEM method in PAMDPs, previous works have only shown its efficiency on either discrete action space or continuous action space, while we’re the first to apply such a method to the parameterized action space. Note that one major modification we make when applying MPPI to PAMDPs is that we keep a separate distribution over the continuous parameters for each discrete action and update them at each iteration instead of keeping just one independent distribution for the discrete actions and one independent distribution for the continuous parameters. In other words, we let the distribution of the continuous parameters condition on the chosen discrete action during the sampling process. We believe this is an important change to make when using MPPI for PAMDPs as we don’t want to throw away the established dependency between the continuous parameter and corresponding discrete action in PAMDPs. In the updated manuscript Section (6.3) we have included a new ablation study result for this change which shows that this change is indispensable for DLPA to do proper planning in PAMDP.

In terms of novelty, we have also added an entirely new theoretical analysis section in the updated manuscript where we derive some performance guarantees / bounds for the proposed algorithm (Section 5). In general, we provide three main theorems that describe the performance guarantees of DLPA from three aspects: 1. A theoretical bound for the cumulative return difference between the rollout trajectories of DLPA and the optimal trajectories. 2. A theoretical bound for the multi-step prediction error of DLPA. 3. A theoretical bound for how the estimation error will change with respect to the number of samples and the cardinality of the continuous parameter space.

Q2: “it would be advisable to provide additional experimental results with a consistent number of training steps for all algorithms to ensure a fair comparison.”

A2: Thanks for your suggestion, before we only show the result when DLPA has already converged in Figure 3. We aim to use Figure 3 to show the superior sample efficiency achieved by DLPA. Table 1 shows the asymptotic performance where all the other algorithms have fully converged. Even after giving all the other baselines a significantly larger number of samples, DLPA still achieves better performance in 6 out of the 8 tested domains and comparable performance in the other 2. We have also updated the plots for the comparison of all methods with full steps in Figure 6 in the appendix, where the results for the baselines are very close to what Li et al. 2022 reported in their paper. Note that all the experiments were run on a Nvidia GeForce RTX 3090 GPU. We directly use the source code provided by the authors for those baselines and so the resources are kept the same for all benchmarks. We also run DLPA for an additional amount of steps after convergence for each environment, and the performance is still quite stable as shown in Figure 6.

Q3: “For the experiments assessing the impact of CEM-based MPC on DLPA, it would be beneficial to include ablation studies with other traditional model-based non-MPC methods, such as MBPO, applied in the context of PAMDP.”

A3: Thanks for your suggestion. We add the comparison to the Dyna-like model-based RL method in Section 6.3, which is a baseline very close to MBPO that does not use MPC. We use a model-free RL framework for PAMDP—PDQN—as the policy learning framework given the data generated from the learned model. We have also tuned the proportion between the generated data and the real-world rollouts to make sure the baseline reaches its best performance. As we can see from the training results on two benchmarks, this non-MPC baseline generally performs better than the model-free baselines but is not as good as DLPA.

Q4: “What specific form do the parameters of CEM take in this context? Are they similar to the parameters of a policy neural network, and how do they relate to the state input?”

A4: The parameters of CEM in DLPA directly control the distribution over the discrete actions and continuous parameters and are different from the parameters for a policy neural network. Specifically, for discrete actions, the parameters of CEM are the parameters for a multinomial distribution. We sample from this distribution to generate discrete action during planning. For continuous parameters, the parameters of CEM are the parameters for a Gaussian distribution. We directly sample from it to generate the corresponding continuous parameters during planning. After randomly initializing the action distributions, we use the learned model and the initial state input to rollout imagination trajectories and estimate their cumulative return. And we pick the actions to execute based on the estimated returns. Note that for each planning step, we will update the CEM parameters for several iterations (described by Eqn 5, 6, 7) given the estimated returns.

Q5: “How does the computational complexity of CEM changes as the action space expands?”

A5: We have included a new Table in appendix 8.6 that compares the clock time of planning and the number of timesteps needed to converge as the action space expands. We tested this on the Hard Move domain, where the number of discrete actions changes from $2^4$ to $2^{10}$ (one continuous parameter for each of them). As shown in the table, while the number of samples increases as the action space expands, it’s still within an acceptable range even when it’s extremely large. We also quantify this with theoretical analysis which is included as a new section in the updated manuscript.

Q: What is the research question you study? In other words, what is the specific difficulty or particular problem when applying existing model-based RL methods on PAMDP tasks?

A: Thank you for the review. The research question of this paper centers on Reinforcement learning problems where actions are of a mixed type and interactions are readily described by Parameterized Action MDPs (PAMDPs). Here we would like to emphasize again that the primary contribution of this paper is the first model-based RL algorithm for the PAMDP setting, which outperforms all the previous model-free RL methods for PAMDP on 8 standard PAMDP benchmarks. PAMDP is a very common problem setting in real-world tasks, and has gained a lot of attention in recent papers despite all of them being model-free methods. We hope our paper can draw other researchers' attention to the model-based aspect of this problem setting since DLPA already performs better than the model-free methods without very fancy designs.

In terms of the specific design when applying model-based RL on PAMDP tasks, besides the multi-step prediction technique which is also mentioned by the other reviews, one major modification we make when applying MPPI to PAMDPs is that we keep a separate distribution over the continuous parameters for each discrete action and update them at each iteration instead of keeping just one independent distribution for the discrete actions and one independent distribution for the continuous parameters. In other words, we let the distribution of the continuous parameters condition on the chosen discrete action during the sampling process. This is an important change to make when using MPPI for PAMDPs as we don’t want to throw away the established dependency between the continuous parameter and corresponding the discrete action in PAMDPs. In the updated manuscript we have included a new ablation study result for this change which shows that this change is indispensable for DLPA to do proper planning in PAMDP (Section 6.3).

In terms of novelty, we have also added an entirely new theoretical analysis section in the updated manuscript where we derive some performance guarantees / bounds for the proposed algorithm (Section 5). In general, we provide three main theorems that describe the performance guarantees of DLPA from three aspects: 1. A theoretical bound for the cumulative return difference between the rollout trajectories of DLPA and the optimal trajectories. 2. A theoretical bound for the multi-step prediction error of DLPA. 3. A theoretical bound for how the estimation error will change with respect to the number of samples and the cardinality of the continuous parameter space.

Q3: “the reported results both in Figure 3 and Table 1 seem to much lower than the results reported in Li et al. 2022 and their HyAR approach. How much resources did you spend training the benchmarks versus your DLPA?”

A3: This is because, in Figure 3, we limit the x-axis, the time steps, to the point where our DLPA has already converged. We aim to use Figure 3 to show the superior sample efficiency achieved by DLPA. Table 1 shows the asymptotic performance where all the other algorithms have fully converged. Even after giving all the other baselines a significantly larger number of samples, DLPA still achieves better performance in 6 out of the 8 tested domains and comparable performance in the other 2. We also show the plots for the comparison of all methods with full steps in Figure 6 in the appendix, where the results for the baselines are very close to what Li et al. 2022 reported in their paper. Note that all the experiments were run on a Nvidia GeForce RTX 3090 GPU. We directly use the source code provided by the authors for those baselines and so the resources are kept the same for all benchmarks.

Q4: “Learning models conditioned just on the initial state and not intermediate transitions must be much more difficult in stochastic environments (either stochastic transitions or actions), have you done any analysis in this direction?”

A4: Thanks for pointing this out. Actually all the 8 domains we tested are stochastic in terms of the transition function. And our ablation studies have shown the superiority of this approach in the tested environments. We have included one more ablation study results plot on the Platform environment in the updated manuscript. As the reviewer mentioned, this approach can be much more difficult in stochastic environments, so we multiply a discount factor along the multi-step prediction in Eqn. (4) to help stabilize training.

Thank you for the thoughtful review. We believe we can address your concerns.

Q1: “The paper brings together PAMDPs and model predictive control in a very conventional way and so I do not find DLPA novel enough from a technical perspective.”

A1: DLPA, while perhaps conventional, is the very first model-based RL algorithm for the PAMDP setting. It outperforms all the previous model-free RL methods - many of which have been published at ICLR, including one just last year - for PAMDP on 8 standard PAMDP benchmarks. The PAMDP setting is very common in real-world tasks, and has drawn a lot of attention in recent papers, but all of them are model-free methods. We hope our paper can draw other researchers’ attention to the model-based aspect of this problem setting since DLPA already performs better than the model-free methods, even the most complex ones.

Meanwhile, we believe our algorithm also has some unconventional techniques which we found to be useful in practice. For example, we use the initial state and intermediate predictions to do model learning. Moreover, for using MPPI for PAMDPs, previous works have only shown its efficiency on either discrete action space or continuous action space, while we’re the first to apply such a method to the parameterized action space. The main modification we make when applying MPPI to PAMDPs is that we keep a separate distribution over the continuous parameters for each discrete action and update them at each iteration instead of keeping just one independent distribution for the discrete actions and one independent distribution for the continuous parameters. In other words, we let the distribution of the continuous parameters condition on the chosen discrete action during the sampling process. We believe this is an important change to make when using MPPI for PAMDPs as we don’t want to throw away the established dependency between the continuous parameter and corresponding discrete action in PAMDPs. In the updated manuscript we have included new ablation study results for this change which shows that this change is indispensable for DLPA to do proper planning in PAMDP (Section 6.3).

In terms of novelty, we have also added an entirely new theoretical analysis section in the updated manuscript where we derive some performance guarantees / bounds for the proposed algorithm (Section 5). In general, we provide three main theorems that describe the performance guarantees of DLPA from three aspects: 1. A theoretical bound for the cumulative return difference between the rollout trajectories of DLPA and the optimal trajectories. 2. A theoretical bound for the multi-step prediction error of DLPA. 3. A theoretical bound for how the estimation error will change with respect to the number of samples and the cardinality of the continuous parameter space.

Q2: “even though the testing environments are used by other previous works, they still seem to be relatively simplistic even compared to other game based benchmarks such as ATARI.”

A2: We first want to point out that all these 8 testing environments are the ones used in an ICLR paper for PAMDP last year. The PAMDP setting is quite recent and so the leading experimental benchmarks are not as complex as the MDP setting. We agree with the review that evaluating and developing our algorithm on more complex PAMDPs is one of our main future directions. But in order to address the reviewer’s concern, we further test our method on a much more complex domain Half-Field-Offence (HFO) domain [1], where both the state space and action space are much larger. Besides, the task horizon is much longer and there is more randomness existing in the environment. HFO is originally a subtask in RoboCup simulated soccer. We are working on this domain and will update the manuscript with the new results soon. Update: we have added the new results in appendix Section 8.7. DLPA still outperforms the other model-free PAMDP baselines in this much more complicated domain.

[1] Hausknecht, M., Mupparaju, P., Subramanian, S., Kalyanakrishnan, S., & Stone, P. (2016, May). Half field offense: An environment for multiagent learning and ad hoc teamwork. In AAMAS Adaptive Learning Agents (ALA) Workshop (Vol. 3). sn.

Thanks for your very positive comments and thoughtful suggestions! We correct the grammar and presentation issues according to your kind advice and address your concerns below.

Q1: From what state are the action sequences sampled?

A1: Action sequences are sampled starting from the current timestep t (so the initial state s_t) until timestep t+H from randomly initialized distributions like Gaussians.

Q2: Suggestions for references, notations and other writing issues.

A2: Thanks for pointing all these out! We have updated the manuscript accordingly.

Q3: “Are your action distributions unconditional on any context?”

A3: At the first timestep of each iteration for planning, we just set the distribution over discrete actions to be a multinomial distribution with parameter \theta_k^0 = 1/K, and set the distributions over the corresponding continuous parameters to be a multivariate Gaussian distribution with parameters \mu_k^0 = 0, \sigma_k^0 = 1. Then for all the timesteps afterward, the action distributions are conditioned on the distributions at the last timestep—weighted by the cumulative return.

Q4: “I need help understanding these results, because they don't seem consistent with Figure 3. In Figure 3., DLPA achieves the highest performance in Platform and Goal. Why then is HyAR bolded?”

A4: This is because, in Figure 3, we limit the x-axis (time steps) to the point where our DLPA has already converged. We aim to use Figure 3 to show the superior sample efficiency achieved by DLPA. Table 1 shows the asymptotic performance where all the other algorithms have fully converged. Even after giving all the other baselines a significantly larger number of samples, DLPA still achieves better performance for 6 out of the 8 tested domains and comparable performance on the other 2.

Q5: “This algorithm seems to use a heavy amount of computation. Can you comment on the algorithm's complexity?”

A5: We have included a new Table in appendix 8.6 that compares the clock time of planning and the number of timesteps needed to converge as the action space expands. We tested this on the Hard Move domain, where the number of discrete actions changes from $2^4$ to $2^{10}$ (one continuous parameter for each of them). As shown in the table, while the number of samples increases as the action space expands, it’s still within an acceptable range even when it’s extremely large. We also quantify this with theoretical analysis which is included as a new section in the updated manuscript.

Thank you for the thoughtful review We address your concern as follows:

Q1: “What is the main contribution behind DLPA, or could authors comment on specific challenges solved when using MPPI for PAMDPs?”

A1: The primary contribution is the first model-based RL algorithm for the PAMDP setting, which outperforms all the previous model-free RL methods for PAMDP on 8 standard PAMDP benchmarks. The PAMDP model is appropriate to many very common real-world tasks, and has gained a lot of attention in recent papers - however, all existing methods are model-free. We hope our paper can draw other researchers' attention to the model-based aspect of this problem setting since DLPA performs better than even the most sophisticated model-free methods. In terms of novelty, inspired by the reviewer’s question, we have added a new theoretical analysis section in the updated manuscript where we derive some performance guarantees / bounds for the proposed algorithm. Thank you for the suggestion!

For using MPPI for PAMDPs, previous works have only shown its efficiency on either discrete action space or continuous action space, while we’re the first to apply such a method to the parameterized action space. The main modification we make when applying MPPI to PAMDPs is that we keep a separate distribution over the continuous parameters for each discrete action and update them at each iteration instead of keeping just one independent distribution for the discrete actions and one independent distribution for the continuous parameters. In other words, we let the distribution of the continuous parameters condition on the chosen discrete action during the sampling process. This is an important change to make when using MPPI for PAMDPs as we don’t want to throw away the established dependency between the continuous parameter and corresponding the discrete action in PAMDPs. In the updated manuscript we have included a new ablation study result for this change which shows that this change is indispensable for DLPA to do proper planning in PAMDP (Section 6.3).

Q2: “What algorithms are used to solve Eq. 1 in terms of model learning? Also, it is unclear why the authors imply that the ground truth state was not used as input to train their model in DLPA. For example, the ground-truth state s_{t+1} is needed to compute the loss L_joint defined in Eq. 1.”

A2: To solve Eqn. (1), we simply learn three neural networks for the reward predictor, transition predictor and termination predictor respectively using supervised learning. And we use the mean square error as the loss of the state and reward prediction, and use cross entropy loss as the objective function for updating the termination signal. We train our model based on multi-step prediction (only the ground truth state at step 1 is needed) to encourage the model to do better H-step long-term prediction. This design for the training process is consistent with the planning process, as H is also the planning horizon in MPPI where we do the same long-term prediction to select actions. As also pointed out by reviewer WRCa, this choice allows gradients to flow back through time and assign credit more effectively than the alternative, which is one-step training. We found this design choice to be especially useful during empirical evaluation as shown in Section 6.3. We have also included one additional experimental results plot on Platform to show this in the updated manuscript.

Q3: “Does DLPA deliver any theoretical properties to justify its performance on PAMDPs? Authors are encouraged to discuss the convergence of parameterized actions from the DLPA framework, if any.”

A3: We thank the reviewer for pointing this out. We have added a theoretical analysis section (also in the appendix) in the updated manuscript (section 5). In general, we provide three main theorems that describe the performance guarantees of DLPA from three aspects: 1. A theoretical bound for the cumulative return difference between the rollout trajectories of DLPA and the optimal trajectories. 2. A theoretical bound for the multi-step prediction error of DLPA. 3. A theoretical bound for how the estimation error will change with respect to the number of samples and the cardinality of the continuous parameter space.