Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control

Weaknesses

Running Policies at High Frequencies in Robotics

While high-frequency hardware may alleviate some challenges, our work addresses scenarios where computational resources are insufficient. These scenarios include low-compute hardware, costly observation processing (e.g., images or LIDAR), and low-bandwidth communication. The existence of high-frequency hardware does not negate the utility of our method, which performs comparably at high frequencies while enabling significant improvements for low-frequency scenarios.

SRL’s ability to function at both high and low frequencies without retraining is a key contribution. As demonstrated, SRL maintains performance at high frequencies and provides significant advantages under low-frequency constraints.

---

Biological Plausibility

Our work focuses on system- and algorithm-level biological plausibility, inspired by action chunking observed in the brain. While mechanistic plausibility (e.g., avoiding backpropagation through time) is a separate area of research, it is beyond the scope of this paper. We have also further removed references to biological plausibility to address this issue

---

Inference Time Discussion

We clarify that reducing inference time is not the primary motivation of our paper. Instead, we aim to address control challenges at low decision frequencies. The inference time of the first action in a macro action is comparable to SAC. However, if subsequent observations are unavailable (due to low sensor frequency or costly observation processing), SAC fails, as demonstrated in our experiments.

If you found text in our abstract or elsewhere that suggested a focus on inference time reduction, please let us know so we can correct it.

---

Claim Regarding Low Control Frequencies

We have added SAC-J (action repeat) as a baseline, and SRL significantly outperforms SAC-J across all environments except Swimmer, where repetitive actions are less detrimental. Additionally, we address the results reported in FiGAR and TempoRL:

FiGAR: Demonstrates lower scores (947.06 on Ant and 3038.63 on Hopper) compared to SRL’s 6500 and 3500 on these tasks, indicating non-competitive performance.

TempoRL: Limited to Pendulum-v0

Furthermore, these results cannot be extended to variable action sequence lengths required in FAS settings. Results from our implementation (based on TD3) (Figure 8) show TempoRL’s average action sequence length is 2.6, far below SRL’s ability to optimize for longer sequences. We hope this clarifies the claim that existing methods are not well-suited for low control frequencies in FAS settings.

---

Discussion of Hierarchical RL

We acknowledge that hierarchical RL offers relevant approaches, such as manager-worker architectures. While SRL could be integrated with such methods, our work focuses specifically on generating temporally consistent action sequences to address low-frequency control. Due to space constraints, we limit our discussion to directly related approaches.

---

Specific issues in text

Line 055: We have supported this claim with experiments (SAC-J) and additional reference.

---

Questions:

In line 098 the claim “Our model learns open-loop control utilizing a slow hardware and low attention, and hence also low energy.” is made. Why should that be true?

Energy efficiency relates to reduced attention and slower hardware requirements, both commonly associated with low-energy systems.

In line 150 the claim that online planning is not feasible in robotics or the brain is made. In my experience this is not true. How did the authors come to this conclusion?

In Appendix Section A.7, we explore the neural basis for sequence learning in humans. Specifically, [2] demonstrates that after training, executing the same action sequences requires fewer neural activations. This reduction in activation suggests that the brain may employ a mechanism akin to SRL, commonly referred to as action chunking. If a model were used for sequence generation after training, the activation levels would remain consistent, which is not observed.

Furthermore, biological neurons typically operate at firing rates of 10–50 Hz. Considering the multiple neuronal layers required to process observations, query a model, and produce actions, it becomes implausible for the brain to rely on high-frequency model queries to achieve temporally precise action sequences. Instead, it likely employs precomputed sequences or learned open-loop strategies.

This principle can be extended to robotics. Current robotics hardware addresses these challenges by leveraging GPUs, high wall power, or large-capacity batteries to maintain high-frequency control. However, adopting principles from SRL could pave the way for robots that are less dependent on costly, fast, and energy-intensive hardware. By enabling efficient control with low-frequency decision-making, SRL offers a potential path toward more sustainable and versatile robotic systems.

Overall, the paper feels like it started out motivated by biological observations, changed its motivation to real-time control on low-resource hardware at some point, and then starts to swerve in the direction of dealing with missing observations in the main text (only in this context averaging over the performance of one policy over a wide range of control frequencies makes sense). This structure is suboptimal, in my opinion.

[1] Allshire, Arthur, et al. "Transferring dexterous manipulation from gpu simulation to a remote real-world trifinger." 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.

Questions:

In line 098 the claim “Our model learns open-loop control utilizing a slow hardware and low attention, and hence also low energy.” is made. Why should that be true?

This is related to the Inference Time Discussion. If the inference time is the same as SAC, then the hardware requirement is the same.

In line 150 the claim that online planning is not feasible in robotics or the brain is made. In my experience this is not true. How did the authors come to this conclusion?

I am still not convinced that this claim holds in this generality. Humans clearly do plan, and so do some approaches used in robotics. Whether this is feasible depends on a lot of factors, for example how reactive the agent has to be or how energy efficient but in general planning is feasible.

The next paragraph states that MPC requires very short timesteps to work. However, that many robotics papers use high frequencies does not mean that it is strictly required. For example, [1] does MPC over skills which has a very low frequency.

We could not find any specific mention of the control frequency in the referenced material. Could you kindly point us to the relevant section or provide additional details? This would help us ensure a more accurate and informed response. Thank you!

In section 3.4 the authors state “Our skill dynamics model requires only 1/H dynamics predictions and action selections of the flat model-based RL approaches […]”. In table 2 in the appendix the value 10 is given for the hyperparameter 10. So MPC runs at a frequency 10 times lower than the MDP frequency.

Would it be possible to visualize the action sequences predicted by SRL compared to those of SAC? We apologize for not explicitly mentioning this in the paper.

The supplementary materials already contain videos showcasing SRL and SAC on the Lunar Lander environment.

Apologies for not being more precise. I intended to suggest that the action sequences themselves could be visualized, for example in a line plot, and contrasted to those SAC chooses.

In line 516 the claim “In deterministic environments, a capable agent should achieve infinite horizon control for tasks like walking and hopping from a single state.” is made. This is infeasible in environments with complex, possibly unstable dynamics since errors do not disappear but can instead compound and blow up. I would recommend revising this paragraph.

We have changed the line to In noiseless deterministic environments

The point about compounding errors still holds. No learned model is perfect so errors in the state prediction will necessarily occur. Depending on the dynamics of the environment, those errors can blow up, rendering the action sequence ineffective (the agent would trip over after a short time). Dynamics which are not stable are known to be common and lead to these issues. I therefore still think what you are intending to do (walking without sensory input) is not possible. (Also see reviewer thGh’s comment on this.)

While much of your review emphasizes robotics, SRL’s applications extend beyond this domain. Examples include: […]

I agree that these points are quite relevant for the motivation of SRL (except for the ‘Low Compute Devices’ for the reasons I discussed above). However, they are not used as a motivation in the paper. Instead, a motivation that, in my opinion, does not correspond to what SRL can actually deliver is used (saving compute in real-time control).

My current assessment of the revised paper is:

• The material present in the paper (method and experiments) is quite interesting and relevant. With the added experiments on SAC-J, this passes the bar for me.
• The motivation for the algorithm is still somewhat misleading, as SRL does not allow running at higher frequencies compared to SAC without introducing a delay (which is not discussed).
• There are still claims in the paper that are, in my opinion, not sufficiently substantiated or even incorrect. For example in line 070: “ Consequently, in embodied agents, all components—sensors, compute units, and actuators—are synchronized to operate at the same frequency. Typically, this frequency is limited by the speed of computation in artificial agents.” To the best of my knowledge, this is not true in robotics. It is quite common that different sensors, the actuators and an RL policy run at different frequencies. As an example, consider [1], where the policy runs at 50 Hz, the actuators at 1 kHz, and the object tracking at 10 Hz. I would advise against making such claims without reviewing the robotics literature first.
• I agree with reviewer YC71 that the overall quality of the paper (positioning with respect to related work, citations for claims, quality of figures) is still somewhat below the threshold for acceptance.

Inference Time Discussion

We clarify that reducing inference time is not the primary motivation of our paper. Instead, we aim to address control challenges at low decision frequencies. The inference time of the first action in a macro action is comparable to SAC. However, if subsequent observations are unavailable (due to low sensor frequency or costly observation processing), SAC fails, as demonstrated in our experiments.

If you found text in our abstract or elsewhere that suggested a focus on inference time reduction, please let us know so we can correct it.

A major part of the motivation of the paper is the argument that high decision frequencies and low reaction times are impractical in real-world settings (e.g. second sentences of the abstract, line 47 ff or line 75 ff in the introduction) due to high hardware requirements. SRL is presented as a solution to this problem. The issues here is that to supply the actuators with input at the required high frequency, SRL would have to produce the first action at an inference time smaller than the inverse of the actuator frequency. Failing to do so would introduce a delay that is not accounted for in the experiments and that would require additional care. In this sense, inference times and actuator frequencies are related. As the first action requires an inference time comparable to SAC, SRL has the same hardware requirements as SAC, contrary to the motivation of the paper.

In my opinion, this is a big issue that is still not addressed in the paper.

Dealing with missing observations during inference is a more suitable motivation for SRL but it is not very apparent in the abstract and introduction. This line of motivation would also require properly discussing related work that deals with missing observations which is not the case so far.

Claim Regarding Low Control Frequencies

We have added SAC-J (action repeat) as a baseline …

Thank you, I appreciate the additional baseline! However, I do not think FAS is a fair metric in this case as SAC-J is trained with a fixed action repeat but is evaluated with different action repeats. When SAC-J is evaluated with the same action repeat, then it seems to be quite competitive in some cases, for example on InvertedDoublePendulum-v2, or even better than SRL on InvertedPendulum-v2 (see Figure 2). If FAS is to be used as an evaluation metric, then training with variable action repeats (as suggested before) would make more sense, in my opinion.

In the plot corresponding to Pendulum-v1 in Figure 2, SAC-4 is compared to SRL-16. This seems particularly misleading because SAC-4 cannot work well for larger action repeats it never saw during training.

FiGAR: Demonstrates lower scores (947.06 on Ant and 3038.63 on Hopper) compared to SRL’s 6500 and 3500 on these tasks, indicating non-competitive performance. TempoRL: Limited to Pendulum-v0

FiGAR: The mentioned values are competitive with the base RL algorithm FiGAR was built on (TRPO). It is not clear how FiGAR would do on these tasks when a modern off-policy RL algorithm would be used instead. FiGAR and TempoRL are the most relevant baselines so it would have been interesting to run them on the considered tasks.

Specific issues in text

Line 055: We have supported this claim with experiments (SAC-J) and additional reference.

The reference to Figure 5 in Dulac-Arnold et al. 2020 does not fit the context of the sentence. The sentence is on reaction times but Figure 5 in the cited arxiv version relates to the observation dimension. The experiments on SAC-J also do not directly relate to reaction times as there is no delay between receiving an observation and outputting the first action.

Small remark

The last two subsection in Section 4 are called “Learning Critic” and “Learning Policy”. I think “Learning the Policy” etc. would sound better and would be consistent with the naming of the subsection “Learning the Model”.

Thank you for your detailed response. It is indeed in the discussion of the method and the presentation of the results where I still see some room for improvement. More concretely, I think the paper could be made more accessible to an audience that is interested in applying SRL on real-world problems. I do not think all references to robotics should be removed. However, I would argue in favor of being careful with general claims given how diverse robotics set-ups and control algorithms are. If the remaining points (addressed below) could be worked in the motivation, the discussion of the algorithm, and the presentation of the results, I would be happy to raise my score.

Discussion about frequency, delays, and inference times

I agree with your argument that predicting actions ahead of time to allow for observation processing or possibly other means of dealing with the delay introduced by the processing of the observation leading to the first action can enable SRL to run with a higher frequency than SAC in practice. For this reason, I think this discussion should be in the paper!

While Section 2 mentions frequency and delay it does so from a more biologically motivated perspective and mostly talks about computation in the brain. For readers that are actually interested in trying out SRL on real hardware, this discussion is not sufficient to point them in the right direction for achieving a high frequency and good performance. I think it might be possible to cut the biological motivation a bit and mention content from your response to me instead. For example, there could be a paragraph on practical considerations somewhere in Section 4.

FAS as a metric to compare SAC(-J) and SRL

Indeed, SAC is not intended to be evaluated in a non-stationary environment. If the transition dynamics at test time differ from those experienced during training time, performance can be arbitrarily bad. However, this is exactly what evaluating SAC(-J) with FAS does, as the control frequency is changed after training. For this reason, I do not think FAS is a relevant metric to evaluate SAC(-J) with. What would be more interesting, is a direct comparison between SAC-J and SRL-J. SRL has the potential to be better here as well, and the interesting question is: How much better is it in practice?

What I meant by training SAC with variable frequencies is not to train it with one frequency after the other but to vary the frequency stochastically during training. This stochasticity could be considered part of the environment which would therefore be stochastic but not non-stationary. Clearly, this setting is very challenging and SAC would probably not do well. However, it would still be a fairer evaluation than FAS applied to SAC with a fixed J.

Reference to back up claim about RL agents struggling with large reaction times

Thank you for the clarification. The Dulac-Arnold reference makes sense to me now and so does Figure 5 in the appendix. I am still wondering a bit if reaction time is the right word here because the RL agent reacts instantly after receiving an observation. Maybe “decision frequency” would be more precise.

Feasibility of online planning

The sentence containing “making it [online planning] nonviable for energy and computationally constrained agents like the brain and robots” in line 136 still sounds like online planning was not feasible for brains and robots. However, it is feasible in many situations, just not when the computation budget is very low or short reaction times are required. I would recommend rephrasing this sentence.

Visualization of predicted action sequences

I was thinking about a simple environment like an inverted pendulum. SRL should anticipate reaching the target configuration which should be reflected in the action sequence, even though it does not receive a new observation that would inform it about reaching it. Showing an easily interpretable quantitative result like this could strengthen the paper and set SRL apart from action repeat but is not strictly required.

Claim about all components of RL systems operating at the same frequency

My issue with this sentence

Consequently, in embodied agents, all components—sensors, compute units, and actuators—are synchronized to operate at the same frequency.

is that it claims that all components in embodied agents operate at the same frequency which they clearly do not. I think a careful rephrasing could avoid misunderstandings here.

For your interest: In the cited paper, the policy is just fed the last available pose estimate for the object. This is not synchronized with the actions the policy outputs. Other parts of the observation (like joint angles, torques etc.) do change at a higher frequency and therefore the actions the policy outputs also change at 50 Hz.

Thank you very much for the revised text, the additional experiments, and the new figures. I really appreciate the effort the authors have put into the paper during the rebuttal.

1. Biological Perspective and Practical Considerations

The revised version of Section 2 is excellent, greatly adds to the motivation of SRL, and clarifies its placement with respect to existing efforts to bring RL to low-compute hardware.

The new section on practical considerations in the appendix is also appreciated.

2. Learning Curve comparisons

The direct comparisons between the learning curves of SAC and SRL are quite interesting and confirm the advantages of SRL over SAC with action repeat. I agree with the authors that there are some environments where repeating actions can improve exploration but it is not necessary for SRL to tackle this challenge as it is more focused on providing high-frequency actions where they are actually beneficial.

I appreciate the added experiment on randomized frame skipping with SAC. The experiments indeed show that this strategy has its limitations and does not scale well to complex environments.

3. Thank you!

4. I appreciate the rephrasing. The statement is more precise now, in my opinion.

5. Thank you for the addition.

6. Comparison with TempoRL

Adding results on TempoRL is a good addition that makes the comparison of SRL to baselines more thorough. I understand that TempoRL is not suitable for the FAS setting but it is still interesting to compare its performance to SRL.

My concerns about the manuscript have been sufficiently addressed, and I find it to be indeed significantly improved. I have therefore raised my score. Thank you for the additional work you put into the submission!

I would add one final remark: I would encourage the authors to reference the new material in the appendix in the main text in some form. It might otherwise be overlooked by the reader.

Thank you for your thoughtful and constructive review. We greatly appreciate your positive feedback on the clarity, motivation, and experimental rigor of our work, as well as your suggestions for improvement. Below, we address the specific weaknesses and questions you raised, along with the changes made to the manuscript.

---

Weaknesses

Sim-to-Real Transfer

While bridging the sim-to-real gap is an important motivation for SRL, its applications extend beyond this scope. As highlighted in the paper, the SRL framework addresses open problems in reinforcement learning, including observational dropout and adaptive control under varying decision frequencies. Below are additional examples of practical use cases where SRL can be advantageous compared to traditional RL methods:

1. Low Compute Devices: Edge devices with constrained computational power often face delays in processing observations (e.g., images or videos). SRL can output multiple actions per observation, compensating for slow processing speeds.

2. Prohibitive Observation Costs: Some applications involve costly or infrequent observations, such as health monitoring where periodic interventions are based on intermittent blood tests. SRL can effectively operate in such settings. This setting was originally proposed in [1].

3. Remote Control with Low Bandwidth: Low-bandwidth communication between sensors and controllers creates challenges for real-time control. SRL can adapt to these conditions by generating sequences of actions between observations.

Action Repetition and Macro-Actions

We have incorporated a version of action repetition into SAC (denoted SAC-$j$) and included the results in the revised manuscript. These comparisons demonstrate that SRL consistently outperforms SAC-$J$ across all action repetition lengths, highlighting the effectiveness of SRL.

---

Minor Errors

Thank you for pointing out the minor errors. We have corrected them.

Finally, I can't say that I fully agree with the statement "simple tasks like walking can be performed without input states if learned properly". Biological agents also use a range of inputs to maintain gait.

We have revised the statement to: "..simple tasks like walking can be performed without input states in noiseless simulations if learned properly."

---

Questions

Q1: Biological Plausibility of Batched Experience Replay

In this work, we focus on system- and algorithm-level biological plausibility rather than mechanistic biological plausibility. While batched updates (like backpropagation) are not biologically plausible, they are a pragmatic choice for scaling and training efficiency in reinforcement learning algorithms. These methods allow us to explore biologically inspired structures and principles (e.g., hierarchical control and variable decision frequencies) without being constrained by low-level biological constraints.

Q2: Explanation of $\alpha$ in Equation 3

The parameter $\alpha$ is the temperature parameter from the original SAC algorithm. It controls the relative importance of the entropy term in the objective function, balancing exploration and exploitation. We have added a clarification to the manuscript.

---

We appreciate your positive assessment and the thoughtful suggestions provided. These have allowed us to refine the manuscript further. The additional experiments, clarified statements, and corrections aim to address your concerns fully. Thank you again for your time and effort in reviewing our work.

Question for the Reviewer

Are there any other concerns that we can address in the current or future revisions?

References

[1] Hansen, Eric, Andrew Barto, and Shlomo Zilberstein. "Reinforcement learning for mixed open-loop and closed-loop control." Advances in Neural Information Processing Systems 9 (1996).

After reading the other reviews and rebuttals, I would like to add some additional points.

While the proposed method was mainly motivated by low compute latency for actuators, the chosen experiments do not require this on modern hardware as reviewer pfCy has pointed out. Here are ideas for further experiments that would convincingly show the need for slow decision frequencies.

• Control from pixels: Processing image data requires extensive computation. Showing that the number of frames that need processing can be reduced dramatically (e.g. only every 16th for SRL-16) may lead to a more convincing argument.
• MPC baseline: MPC too requires heavy computation. Showing how SRL compares to MPC would further illustrate the advantages of the method. Similarly, using the same kinematic model for SRL is place of the learned one, as suggested by reviewer Y33V, would make a very interesting experiment.

As a final note, I have to admit to be not very knowledgeable in this area of RL that encompasses "frame-skipping, action repetition, long-horizon exploration, and action correlations". Reviewer YC71 is lamenting a lack of discussion of the relevant literature. I can only agree in this regard and would have enjoyed learning more from a more nuanced section 3.3. Nonetheless, the Related Work section in the current manuscript is extensive as is and covers a broad range of backgrounds.

These were some additional remarks to hopefully help deciding on this borderline paper. Rest assured that I will leave my current rating untouched.

Thank you for your detailed and insightful review. Your feedback highlights several important points, and we have made significant improvements to the manuscript based on your suggestions. Below, we address your concerns, describe changes made, and provide clarifications where necessary.

---

Weaknesses

Expanded Baselines and SAC-J

We have added SAC-$J$ as a baseline in the revised manuscript. Our results show that SAC-$J$ significantly underperforms SRL across all environments, except for the Swimmer environment, where action repetition offers a significant advantage due to better exploration. This comparison strengthens the argument for SRL’s efficacy in the FAS setting, particularly when adapting to low decision frequencies.

Generative Planning Method (GPM)

We have also included GPM as a benchmark and observed that SRL significantly outperforms it across tested environments. While additional model-based and model-free methods could provide further perspective, their inclusion is computationally prohibitive. SAC models alone required approximately 40 GPU-days for training in the presented experiments, and expanding the benchmark suite would result in months of computational time. Not to mention that FAS calculation requires 800 evaluation episodes per environment. We believe this acts as an implicit gatekeeping mechanism and should not detract from the significant improvements demonstrated by SRL. Additionally, most existing methods lack mechanisms to adapt to observational dropout or low decision frequencies, making their relevance to the FAS setting limited.

Ablation with Ground-Truth Dynamics

We agree that an ablation with ground-truth dynamics could provide valuable insights. However, this would require reimplementation of algorithms with parallelized environment simulations to support batched updates. We leave it to future work.

Other points:

Thank you for suggesting the Dulac-Arnold et al. paper ([1]). We have added it to our discussion, and it partly addresses concerns about baseline selection. Notably, Challenge 7 in the paper demonstrates using preliminary results that D4PG, like SAC, suffers significant performance degradation at longer timesteps, further supporting the value of SRL for adaptive frequency control.

The statement about "model complexity of zero after training" has been revised for clarity. This refers specifically to model-based online planning methods, where a learned model is required at deployment. SRL’s ability to generate action sequences without a model after training distinguishes it from these methods. Model-based data augmentation methods, while having a model complexity of zero after training, do not provide mechanisms for generating sequences of actions based on a single state.

Other Revisions

Lines 204, 49, and 265 have been updated for clarity. The repetition in the abstract has been removed.

Thank you for your detailed and thorough review. Your comments were invaluable in improving the clarity, rigor, and presentation of our paper. Below, we address your concerns point by point and outline the changes made in the revised manuscript.

---

Lack of Related Works

We appreciate the reviewer’s suggestions for related works and have incorporated them into the revised manuscript. Below, we address their relevance and relation to our work:

Frame-Skipping and Action Repetition: As discussed in Section 3.4, these methods are indeed relevant. Traditionally frame-skipping and action repetition have been studied as methods for better exploration or explainability. In this work however, we suggest that they might be used in place of macro-actions in the FAS setting to act even when the observation is missing.

Discussion of Suggested Works:

[1, 2, 3]: These works focus on exploration techniques (e.g., smooth exploration and temporally correlated exploration). However, they do not address the unique challenges of the Frequency-Averaged Score (FAS) setting, where decision frequency is lower than action frequency. Specifically: [1]: Generalized state-dependent exploration (gSDE) improves exploration and smoothness but relies on continuous state inputs. In FAS settings, it will have to default to action repetition. While gSDE could enhance SAC exploration during training, it does not address evaluation under observational dropout or lower decision frequencies.

[2, 3]: Similarly, these methods leverage temporal correlations to improve exploration but cannot adapt to scenarios with missing observations during evaluation or low decision frequency. [4]: This work (Generative Planning Method, GPM) is indeed closely related. We thank the reviewer for pointing it out and have now included GPM as a benchmark in our experiments. While GPM is primarily focused on exploration, its similarities to SRL make it a valuable comparison. Key differences include:

Objective: GPM is designed to enhance exploration, whereas SRL focuses on precision and adaptability under low decision frequencies.

Training: GPM trains Q-values for sequences without addressing the instability caused by deeper actions diverging. SRL resolves this issue through model-based training.

Evaluation: GPM's reported results do not evaluate full plan execution under observational dropout.

We have now benchmarked GPM in FAS settings, where it performs poorly compared to SRL and even SAC. We hope these clarifications and additional experiments demonstrate our consideration of related works and reinforce the novelty of SRL.

---

Technical Issues

Lines 252–259: We agree this section was unclear. The statement has been corrected. Specifically, SAC automatically tunes the entropy parameter for all actions, making it desirable in our setting as it avoids extensive hyperparameter tuning across different sequence lengths.

Lines 291–296: We clarify that our discussion of credit assignment refers specifically assigning credit to primitive actions in an action sequence performed based on a single state, which differs from the settings addressed in [5]. Thus, [5] does not directly apply to our work.

Line 93: Based on your feedback, we have revised the text to focus solely on catastrophic failure due to missing inputs, which our method explicitly addresses.

---

Poor Experiment Quality

Additional Benchmarks: We have added GPM as a benchmark across four shared environments. As reported, GPM performs poorly in FAS settings, further validating SRL’s effectiveness. Additionally, we included results for SAC trained with different timesteps (SAC-$J$) for a comprehensive comparison.

Performance Clarification: We acknowledge the confusion caused by our presentation of learning curves. In the original Figure 5, SRL and SAC were evaluated under different conditions, making direct comparisons difficult. To address this:

We have revised the manuscript to include separate plots for SAC-$j$ (Figure 3) and SRL-$𝐽$ (Figure 4). Similarly, we have separated the plots for action sequence length (Figures 5 and 6). We clarified that SRL’s learning curves reflect a disadvantageous evaluation condition (state provided every $J$ steps), while SAC (SAC-1 in the revision) is evaluated under standard conditions. For a more equitable comparison, we direct the reviewer to Figure 5 and 6 (in the revision), where SAC and SRL are evaluated under identical settings. In these plots, the left-most markers represent the evaluation under the standard setting (i.e., a state is provided at every timestep). By comparing Figures 3, 4, 5 and 6, it becomes evident that learning curves do not fully capture SRL's performance. For example, while SRL-16 appears to exhibit near-zero performance on InvertedPendulum-v2 in Figure 4, Figure 6 shows that SRL achieves optimal results when tested under the standard setting.

This comparison underscores that SRL does not underperform in any environment when evaluated under standard conditions. However, it is important to emphasize that the primary goal of our work is not to enhance performance in standard settings, as SAC already achieves optimal or near-optimal performance across most environments (except Swimmer). Instead, the focus of our work is to improve performance under the remaining 15 markers, which correspond to settings where decision frequency is reduced, and observational dropout occurs. In these scenarios, our algorithm demonstrates substantial improvements over SAC. Additionally, we have amended the text in the manuscript to provide clearer context and avoid misinterpretations.

We hope this explanation, along with the updated figures and text, adequately addresses concerns regarding the significance and interpretation of the results presented in our work.

---

In 5 out of 10 tasks (InvPendulum, Hopper, InvDPendulum, Reacher, and Swimmer), the Online Planning method outperforms SRL, contradicting the description in the caption of the "second Table 3" on page 9.

We believe that the reviewer is referring to the following statement: “We see that SRL can learn action sequences that perform better than model-based online planning in many environments.”

We have changed it to:

“We see that SRL can learn action sequences and is competitive to model-based online planning.”

---

Excessive Discussion of Biological Concepts

Thank you for this suggestion. The biological discussion has been moved to the appendix to streamline the main paper and maintain focus on the RL contributions.

---

Minor issues:

We have revised the notation for macro actions to $m$ Formatting errors have been corrected.

---

Answer to Questions:

We have addressed the primary concerns regarding related works, benchmarks, and experimental clarity. Additionally, we emphasize that the FAS metric measures average performance across frequencies and was not specifically designed to favor our method.

---

We appreciate your assessment and the thoughtful suggestions provided. These have allowed us to refine the manuscript further. The additional experiments, clarified statements, and corrections aim to address your concerns fully. Thank you again for your time and effort in reviewing our work.

Questions for the Reviewer

1. Have the additional experiments and clarifications addressed your concerns? Are there other related works we may have missed that would strengthen the manuscript?

2. If you are unable to change your rating, could you kindly highlight any remaining issues so we may address them in future revisions?

3. We respectfully disagree with the notion that the necessity of low-frequency RL agents is unproven or unnecessary in robotics. Modern robot hardware has indeed been designed with existing algorithms in mind, which assume fast decision frequencies. This creates a "survivor bias," where only systems that fit these assumptions are published or widely adopted. However, this does not diminish the need for solutions like SRL.

   For instance, [7] highlights challenges similar to those addressed by SRL, including degraded performance at lower frequencies. Additionally, [8], a GitHub repository (albeit 5 years old) for a line-following robot using a Raspberry Pi, provides a concrete example of how limited processing speed hinders RL deployment on affordable hardware. The repository states:

   The Raspberry Pi 3B+ could only run the PPO algorithm at about 1 FPS. At this frame rate, the donkeycar couldn't get around the track much faster than the OpenCV line-following algorithm.

   Further evidence of failures of RL algorithms on affordable and slow hardware is sparse, as negative results are seldom published.

   While SRL might not completely solve such issues, it offers a substantial improvement over existing methods like PPO by enabling effective control at lower frequencies. For instance, at 1 Hz, SRL achieves a reward of -400 on Pendulum and 3300 on Ant, compared to SAC’s -750 and 1400, respectively. These results demonstrate SRL’s superior performance in low-frequency settings. This provides a strong motivation for further exploration of this setting.

   Addressing the choice of Gym environments: Our primary focus was to establish benchmarks on environments that are particularly challenging to solve using extended actions. Notably, previous works such as TempoRL, GPM, NDP, and FiGAR have not addressed difficult environments like Humanoid, Ant, Walker, and Half Cheetah. This limitation can be attributed to two main factors:

   a. Unlike simpler robot arm environments, these locomotion tasks demand precise balancing, which requires high-frequency action updates due to their fast dynamics. As a result, these environments do not benefit significantly from action repetition approaches.

   b. Multiple Action Dimensions: Multiple action dimensions are difficult for open-loop action repetition since it requires the repetition to be synchronized across multiple degrees of freedom which is unsuitable for tasks like locomotion.

4. Our algorithm addresses scenarios where the optimal timestep is not feasible. However, since it is built on SAC, it requires SAC to succeed for at least one choice of timestep. If there is an environment where SAC fails across all timestep choices, SRL would also fail.

   It is worth noting that the default timestep varies significantly across the environments we tested and is typically chosen based on its optimality for traditional RL setups. For example, the default timestep is 8 ms for Hopper and Walker and 50 ms for Ant and Half Cheetah. SAC exhibits poor performance on Hopper and Walker at 50 ms, whereas SRL demonstrates near-optimal performance.

Thank you once again for your thoughtful remarks and the opportunity to improve our work. While we may differ on certain aspects, your comments have been instrumental in refining the manuscript and addressing key gaps in our results and motivation. Your feedback has been invaluable, and we sincerely appreciate the time and effort you have dedicated to reviewing our work. Thank you once again for helping us improve our contribution to this field.

References:

[7] G. Dulac-Arnold, N, Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. "An empirical investigation of the challenges of real-world reinforcement learning." arXiv, 2020.

[8] https://github.com/downingbots/RLDonkeycar

To further address the concerns raised by reviewers YC71, Y33V, and pfCy, we have made the following updates to our manuscript:

-Motivation Concerns: We have clarified and strengthened the motivation behind our work to address the points raised.

-Results on TempoRL: We have added results for TempoRL to provide a more comprehensive comparison with existing methods.

-Comparison Plots for SRL and SAC on the Same $J$: We included direct comparison plots of SRL and SAC on the same evaluation metric to enhance clarity and enable better assessment of their performance.

-SAC with Randomized Frame-Skipping: We provided results for SAC with randomized frame-skipping across three environments, adding depth to the discussion of alternative approaches.

We believe these updates have significantly improved the quality and rigor of our paper. We sincerely thank all reviewers for their constructive feedback and exemplary participation throughout this discussion phase.

If there are any remaining concerns or further clarifications needed, we would be happy to address them.