Achieving Precise Control with Slow Hardware: Model-Based Reinforcement Learning for Action Sequence Learning

Thank you for your taking the time to read our work and for detailed review of our work.

1. We demonstrate this in the current work in Figure 3. SAC fails at human-like reaction times. We have updated the text to say that we demonstrate this in experiments.

2. We provide a control solution for the setting: slow compute frequency but fast actuation and control frequency. We assume no delay in perception or actuation. For RL algorithms, reaction time can be thought of as the time between each reaction (output) of the agent or simply as time between each input taken by the agent since there is not delay assumed and each input produces an output. Additionally, delay can be handled in RL by setting the timestep equal to the delay and appending the last action to the state-space, thus making the MDP stationary again [1]. In this setting, the delay is then equal to the timestep. Therefore, we use timestep as the reaction time for RL. To avoid any confusion, we have removed any mention of latency from the abstract. We provide references in our paper to cite the average human reaction times and the RL control frequencies to reinforce our claim that RL control frequencies are faster than the average human reaction times.

3. The model is deterministic. Line 163 is a typo and we have fixed it.

4. There is a typo on line 192. It should be $\tilde{a}\_{t:t+J}$ $\sim$ $\pi\_\phi(s_t, a\_{t-1})$. Therefore, the actions come from the probabilistic policy $\pi_\omega$. The states $\tilde{s}\_{t+1:t+J}$ come from iteratively running the model on $s_t$ and the actions produced by $\pi\_\omega$. There is no policy $\pi\_\psi$. The parameters $\psi$ are for the critic network. The critic network does not require the previous action in order to output the q-value of the state-action pair. The previous action is required since HSP uses a recurrent network and the action acts as the recurrent connection. In response to this question, we have fixed all the typos within the equations and added detailed explanation of the temporal recall mechanism.

5. We did not run the ASL experiment for other environments as the performance was worse than the normal variant.

6. We are assuming that Fig. 6 is a typo. In Fig. 2, HSP does outperform SAC on 3/6 environments and indeed it is a surprising result. We used the same hyperparameters as the SAC paper for both SAC and HSP and did not do hyperparameter tuning beyond the actor-update-frequency parameter for HSP. SAC is evaluated without any action repetition or delay or any sort. HSP is evaluated on ASL = $J$ so that it sees input every $J$ steps. Therefore, indeed HSP is at a disadvantage. We hypothesize that optimizing multiple future actions based on a single state results in reduced preference for myopic policies that maximize the value of the next state but result in a decreased overall reward. This effect is prominent on the Hooper-v2 task where the HSP-4 outperforms even HSP-2. In response to this question, we have added this insight to the updated revision. We also note that while it might seem like HSP beats SAC on the Humanoid task, but when evaluated SAC is marginally better than HSP. For instance, HSP-2 evaluated at ASL of 2 has an average reward of 8852.38 while SAC has an average reward of 9006.25.

7. We do not introduce repetition during learning because then we will be evaluating one policy of HSP against many different policies of SAC (one for each ASL). Though even in this setting HSP beats SAC, and in response to this question we have added the results for it in the appendix. We wanted to demonstrate the setting of a single policy being able to adapt to missing inputs (at different frequencies). Before training, the frequency modulation can be achieved either by action repetition or by changing the timestep. However, after training, we can only use action repetition.

8. For the model-based online planning, we use the HSP policy and the model trained during the training of HSP. We use a simple online planning algorithm: for any ASL, we feed the first action produced by HSP to the model to get the next predicted state. This state is then fed into the HSP to produce the next action. This is repeated until the number of actions is equal to ASL. All the actions in the action sequence are then performed in the environment. Regarding lines 104-105: It is true that our method is more related to model-free approaches. Therefore, we compare to SAC. However, we did not find any previous work that demonstrate model-free sequence generation. Therefore, we mention model-based online planning in related work.

9. As mentioned in the paper, generative replay greatly benefits the from replay in the latent space rather than the input space. This is because the impact of poor precision of replay in latent space is lower than in input space. Thus, latent space replay could also be introduced to HSP. In this work, we implemented a simple generative replay based on the TCRL work as a preliminary work in this direction. Remarkably, on the walker environment, latent space replay demonstrates very strong results where even when evaluated at ASL-16, HSP-16 produces competitive results to SAC that is evaluated on ASL-1. We present this result to encourage research and to further increase the benefit of our work to the community. While we performed limited experiments and hyperparameter tuning on the latent space networks, we still present the results on all environments for completeness. While we have added the above explanation to the paper.

Additionally in response to the weaknesses and minor points, we have added a definition of macro-actions, added hierarchical RL and movement primitives discussion to 3.2, fixed the typos mentioned and changed the citations to square brackets. Thank you for the retailed comments.

[1] Chen, Baiming, et al. "Delay-aware model-based reinforcement learning for continuous control." Neurocomputing 450 (2021): 119-128.

Thank you for the detailed review and for acknowledging the novelty and the performance of our method. We address weaknesses not already addressed in the questions first:

2. The novelty of our approach is the temporal recall mechanism and the introduction of simultaneously trained model using delayed updates. In response to this, we have updated the sections to highlight the contributions and novelty of our work in the updated revision.
3. In response to this, we have added a definition of macro-actions and elaborated on the scalability issues of existing approaches. We have removed the word ‘principles’ since it is not required and does not add anything to the sentence.

Questions:

1. We follow the format of SAC papers that provide plots and not tables [1,2]. The main improvements of HSP can be seen in Figure 3. However, if the same information is provided in numerical format, it would require a table with 16 rows and two columns for each environment. But, we can release an excel sheet with the raw data required to recreate the figures presented in the paper. If the reviewer can specify for which result they would like numerical comparisons, we will be happy to provide them. The numerical performance gains of HSP over SAC can be summarized in table 2 provided in the pdf. The table shows the longest action sequence length that produces greater than 50% of the best performance on each environment (higher is better). Here, we pick 50% as a threshold for catastrophic failure, however, the threshold can be different for different applications and environments.

2. That is a good question. Since the model is trained in parallel to the actor and critic, the during the initial training, any model predictions are highly inaccurate. However, the model improves with training. We find that the final model accuracy has a direct impact on the performance of HSP for longer ASL and is a better predictor for performance than the complexity of the environment. For example: the final mean squared error for the Half-Cheetah task that has 17 dimensions is around 0.05 while that of Ant is around 0.002 even though it has 27 dimensions. As a result, HSP can maintain performance for longer ASL on Ant than Half-Cheetah. In future, we will focus on improving the accuracy of the model to further improve the results. In response to this question, we have added a discussion on the model accuracy and HSL performance in the updated revision.

3. We believe that HSP will have broad real-world applications. Reinforcement learning algorithms are powerful, however, in the current RL setting the observation, compute and action frequency all utilize the same timestep and need to be synchronized. This limits its practicality as it requires fast costly hardware to be implemented. Additionally, there are many applications might be beyond the reach of even the fastest computers. For example, a recent work demonstrated control of tokamak plasmas using deep-RL [3] that might help accelerate fusion research. However, in their work, they required a control frequency of 10 kHz. This also means that the required compute frequency is 10kHz which might require costly dedicated hardware to achieve. However, with HSP, the compute frequency can be lowered while maintaining the high control frequency as demonstrated in our work. Thus potentially reducing the costs significantly. Similarly, other applications include robotics, drone control (125Hz or 8ms [4]) and any other application that utilizes RL. In response to this question we have added a detailed discussion on the real world applications of HSP to the updated revision.

4. Lower computation can be quantified by calculating the multiply accumulate operations (MACs) for the number of actions produced.

In our current work, we did not optimize the hidden units hyperparameter for the GRU and linear layers in HSP. We picked the same hidden units for all neural networks across all base lines (256 hidden units). As a result, currently, HSP is more computationally intensive than SAC. But it is still superior to online planning since it does not use the model during inference. In the table 1 of PDF: we provide the average (Multiply-accumulate operations) MACs for different number of actions produced by each benchmark.

Scalability can be quantified by robust performance on difficult RL environments. The difficulty of the environments can be quantified by their action and state-space dimensions, and the performance SOTA algorithms on it. Current macro-action approaches do not scale well to difficult continuous control tasks like the Ant, HalfCheetah and Humanoid and thus demonstrate a poor performance on it.

In response to this question, we have added the compute table to the appendix and an explanation on scalability has been added to the updated revision.

Limitations:

1. We believe question two answers this limitation to some extent. However, in response to this, we will add an updated plot for model-accuracy vs. performance (largest ASL above a threshold) in the appendix of the paper.
2. As mentioned above, the scalability of HSP depends on the accuracy of the model and the scalability of the SAC algorithm which HSP utilizes. In response to this, we have added a discussion on the topic to the paper.
3. We have fixed the typo. Thank you for pointing it out.

[1] Haarnoja, Tuomas, et al. "Soft actor-critic algorithms and applications." arXiv preprint arXiv:1812.05905 (2018).

[2] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

[3] Degrave, Jonas, et al. "Magnetic control of tokamak plasmas through deep reinforcement learning." Nature 602.7897 (2022): 414-419.

[4] Kaufmann, Elia, et al. "Champion-level drone racing using deep reinforcement learning." Nature 620.7976 (2023): 982-987.

Thank you for the detailed review and your acknowledgement of the promising direction and applications of our work.

Weaknesses:

Framework performance: We understand that these concerns are based on figure 2. During training, we evaluate HSP-$J$ on action sequence length of $ASL=J$. Therefore, the HSP-$J$ learning curves are evaluated at a disadvantage: receiving input only every $J$ steps. This is more apparent when Figure 2 and Figure 3 are viewed together. For instance: On Ant-v2 environment, Figure 2 demonstrates HSP-16 average reward of around 4000 since it is evaluated on ASL of 16. However, when it is evaluated on ASL of 1, the avg. reward is 5405.75. While this performance is still lower than that of SAC (7073.27), when compared at higher ASL, the performance is significantly better than SAC for all environments and all $J$. Again, looking at Ant-v2, at ASL of 2, HSP-16 has avg. reward of 5726.08 while SAC has a avg. reward -186.15. Thus, HSL trades-off the avg. reward at 50ms (by 23.57%) to not fail at 100ms. We believe that this demonstrates the significant benefit of HSP over existing methods.

SOTA algorithms in RL focus on performance while disregarding other factors. Therefore, all the hyperparameters including timestep are picked to maximize performance. On the other hand, HSP provides an alternative algorithm that can be robust to changes in observation frequency or compute frequency.

Latent space variant: We used the same method as the TCRL paper without any modifications. We presented the Walker-2d results as an encouraging example for further research and to demonstrate that further improvements are possible, thereby increasing the benefit of our work to the community. We understand that we did not clearly state this in the paper and we have added a clarification text in response to your comment. We are willing to move the latent space variant to the appendix and mention it in future work.

Novelty: We respectfully disagree on this point. The novelty of our work is not the production of action sequences but the production of action sequences without utilizing a model. As the reviewer rightly points out, there exists vast amounts of works including MPC and model-based RL that utilize action sequences. However, our work has more similarity to model-free RL than any other area. HSP is primarily a reinforcement learning algorithm based on SAC and does not start with any trained model or system knowledge. Once trained, again, it does not need any model for implementation. In comparison, MPC requires an accurate dynamics model of the system. This limits its practicality to systems that have already been sufficiently modeled. On the other hand, HSP makes no such assumptions. Thus, it is not possible to directly compare HSP to MPC systems. Finally, like existing RL approaches, current MPC approaches require super-human compute frequency to operate well. Examples:

1. Galliker, Manuel Y., et al. (2022) walker at 10 ms.
2. Farshidian, Farbod, et al. (2017) four legged robot at 4 ms.
3. Di Carlo, Jared, et al. (2018) MIT cheetah 3 at 33.33 ms.

Model-based RL: MBRL methods either require a pre-trained model of the environment or require a model during inference or both. Thus, it is not directly comparable to our method and HSP. However, we provide the comparison to online planning as it demonstrates an interesting property of HSP. The performance of online planning depends on the policy and the accuracy of the model. Planning longer sequences requires a model of very high accuracy to reduce the effect of additive noise. The results demonstrates that the effect of additive noise does not affect the performance of HSP to the same extent as online planning. We do not utilize any noise reducing algorithm or SOTA approaches that improve online planning performance since they are not in the scope of our work. In response to this, we have added a detailed explanation of the setting and the results for the online planning experiment.

We understand that the word hardware in the title might cause confusion. We simulated the slow hardware and presented a biologically plausible mechanism for the slow biological hardware. We will change the title of the paper to avoid this confusion. We agree that hardware implementation is a great future demonstration. We are planning to implement it.

Questions:

1 & 2. Like any RL algorithm, the final performance of HSP will depend on the type of stochasticity in the environment. If the stochasticity can be predicted using states and actions and managed by actions of the agent, then $J$ can be larger. However, unpredictable stochasticity can be treated similar to a noisy model. Larger $J$ will lead to poor performance. We believe that the best solution, as you also suggest, is to develop a method to adapt the $J$ and ASL based on the expected noise and stochasticity. This proposed solution is similar to the attention mechanism in the brain and an important future direction of our work. In response to this question, we have added a discussion on stochasticity and added adaptive ASL as future work.

3. In our current work, we did not optimize the hidden units hyperparameter for the GRU and linear layers in HSP. We picked the same hidden units for all neural networks across all base lines (256 hidden units). As a result, currently, HSP is more computationally intensive than SAC. But it is still superior to online planning since it does not use the model during inference. We provide the average (Multiply-accumulate operations) MACs for different number of actions produced by each benchmark in the pdf attached as a measure of computational cost. In response to this question we have added the table to the appendix.

4. We provide the relevant details in the Appendix A.2

We would also like to thank the reviewer for their detailed read of our paper and have fixed the typos pointed out by them.