EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we address each concern:

do the environments considered for the evaluation have the characteristics requested by the constructive proof of prop. 2.1?

Thank you; Prop. 2.1 formally motivates that certain CTMDPs can yield higher returns by acting at higher frequencies. To concretely illustrate this, we use two safety-critical environments:

1. Safety Critical Reacher (Sec. 5.2, Table 4; App. B.1)–Here, unmodeled collision detection requires rapid reaction to avoid a penalty. Policies with slower control frequencies respond too late and accumulate significant penalties, leading to substantially lower returns.
2. Safety Critical Halfcheetah – We created a new environment that likewise penalizes collisions that need immediate corrective actions. We observe that policies acting faster achieve higher returns, consistent with Prop. 2.1– https://imgur.com/a/LNBypMy.

The authors should provide better context, because it is counterintuitive that locomotion, pushing or balancing would benefit so much from more frequent control....results from standard algorithms should be used to put these results into perspective

All our benchmark environments are non-standard because we increase the control frequency to 500Hz (episode length = 1000 steps, i.e., 2 seconds of real time) and remove the typical control-cost term. Each original Gym MuJoCo environment is typically controlled at 12.5–100Hz. Therefore, our modified settings differ substantially from standard benchmarks. We made these changes to research within robotics, the question of whether we can remove the lower-level fixed controllers and instead directly control the motor torques at high frequency. What benefits (e.g., faster reactions, finer control) this higher-frequency control can enable, while revealing the challenges standard learning algorithms face in these now longer-horizon tasks. EvoControl addresses precisely these high-frequency demands.

• Table 3 (main paper) reports relative policy performance at 1M high-level steps.
• We include explicit non-normalized results of Table 3 in Appendix J.7, Table 25, alongside extended training for all baselines up to 1 billion steps in App. J.6.

We now include this in Sec. 5.

What does EvoControl discover that makes these robots run at unprecedented speeds?

In these high-frequency environments, standard policy learners struggle with exploration over long horizons (Peng, 2017)—EvoControl matches the exploration efficiency of temporally abstract goal-reaching methods (Fig. 2) and converges faster to high-reward policies (App J.10, Figure 18). Rollouts (App. J.10, Fig. 18) show EvoControl often applies maximal torques to reach goals quickly while retaining precise control. This likely arises as direct high-frequency policy learning algorithms struggle with less efficient exploration and slower convergence (Figure 2, Sec. 5.2), and due to this, can get stuck in suboptimal policies (L 238, Left Col.) (App. J.6 & J.4), and low-frequency policies using a lower-level goal based policies struggle to learn accurate yet refined high-frequency motor movement, and rely on tuned fixed low-level controllers. We now include this in Sec. 5.2.

results from standard algorithms should be used to put these results into perspective ... consider pretrained policies

Thank you; we now include four additional standard baselines using publicly available Brax code of Soft Actor-Critic and PPO, for both high and low frequency control. We observe that EvoControl still outperforms, https://imgur.com/a/yVFuNKk. Pretrained policies for standard MuJoCo tasks cannot be used because our tasks differ significantly (see above). Instead, we use official high-performance Brax code to ensure the baselines are well-optimized.

experiments on a real robot are only presented in the supplementary material

We appreciate this point and will incorporate real-robot results directly into the main paper (using our extra page) to showcase practical feasibility of EvoControl.

justify why they do not compare to ... hierarchical RL

In hierarchical RL, the most relevant method is HIRO, an off-policy method that assigns state-based goals to a low-level policy, rewarded by sub-goal completion. This closely resembles our “fixed-controllers”, which also receive state-based goals (L192). However, as shown in Table 4, sub-goal methods cannot handle the high-frequency reactions needed in safety-critical tasks. HIRO can also be unstable; with the authors’ hyperparameters and our training budget, it failed to converge and performed near-random, highlighting the need for extensive tuning.

MuJoCo Brax

Yes, MuJoCo with GPU acceleration (MuJoCo XLA (MJX)).

---

We hope that all of the reviewers’ concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we address each concern:

not clear whether it directly suits control applications. Specifically, what is the inference speed of the low-level policy? Can it be deployed at its desired frequency in the real world?

Thank you for highlighting this point. We confirm that EvoControl can indeed be deployed at its desired frequency in real-world settings. In Appendix J.14 (p. 51), we provide a real-robot validation using a 7-DoF Franka Emika Panda on a tabletop manipulation task, where the high-level policy operated at 10Hz and the low-level policy at 200Hz in a zero-shot sim-to-real transfer, https://imgur.com/a/Eq2Dewy, https://imgur.com/a/9yUONbW.

We measured the low-level network’s inference time to be 64 microseconds ($\mu$s) on average (over 1,000 samples), corresponding to a potential 15 kHz operating frequency—well beyond typical robotic control rates. Our real-world experiments yielded three main findings:

1. Stable and Fast: The low-level network infers rapidly (64 µs per step), comfortably enabling deployment at 200 Hz or higher. In practice, the main bottleneck would typically be observation latency (on the order of a few milliseconds). Thus, the low-level controller itself is not the limiting factor for overall reactivity.
2. Better Collision Handling: Compared to a tuned PD controller, EvoControl produced lower contact forces, highlighting the benefits of learned high-frequency torque control for safety.
3. No Fine-Tuning Required: The policies trained in MuJoCo (XLA) transferred directly to the real robot without additional adjustments.

These results strongly support EvoControl’s practicality in real-world environments, clearly demonstrating that the high-frequency policy can indeed be deployed at the desired rate despite practical observation latency constraints.

For Table 3, it is better to report the raw returns without normalization, so that readers can better compare the results with other methods on the benchmark.

Thank you; that is a great suggestion. We already include the raw returns without normalization for Table 3 in Appendix J.7, Table 25.

Allow us to kindly clarify that all our benchmark environments are non-standard because we increase the control frequency to 500Hz (episode length = 1000 steps, i.e., 2 seconds of real time) and remove the typical control-cost term (Appendix E). Each original Gym MuJoCo environment is typically controlled at 12.5–100Hz. Therefore, our modified settings differ substantially from standard benchmarks. We made these changes to research within robotics, the question of whether we can remove the lower-level fixed controllers and instead directly control the motor torques at high frequency. What benefits (e.g., faster reactions, finer control) this higher-frequency control can enable, while revealing the challenges standard learning algorithms face in these now longer-horizon tasks. EvoControl addresses precisely these high-frequency demands.

I think experiments on similar safety-critical tasks with more complex dynamics would be more valuable for practical use.

Thank you for the helpful suggestion. In response, we introduce a new “Safety Critical Halfcheetah” environment with six degrees of freedom and an observation dimension of 19, exhibiting significantly more complex dynamics than our previous 1D safety task. Specifically, we add a blocking wall in 25% of the episodes, with collision force only observable upon contact. This wall blocks the cheetah’s forward path and incurs a penalty for any impact. As a result, the policy must quickly retreat if it detects a collision; otherwise, it continues forward. This design underscores the importance of higher-frequency actions for fast responses in unmodeled safety-critical situations, following a similar setup to "Safety Critical Reacher". Our full results (at https://imgur.com/a/LNBypMy) show that EvoControl still outperforms the baselines in this more challenging safety-critical scenario.

In line 83, CTMDP occurs without explanation.

Thank you for this typo; line 83 now reads "continuous-time Markov decision processes (CTMDPs)".

---

We hope that all of the reviewers’ concerns have been addressed and, if so, they would consider updating their score. If any issues remain, we would be glad to discuss them further, especially in light of your current evaluation.

We thank the reviewer for their thoughtful and constructive feedback. Below, we address each concern:

authors do not really motivate why such MDPs exist, and why they are important

(A) Why Such MDPs Exist

Our Proposition 2.1 is meant to show that there are environments/continuous-time MDPs in which acting at higher frequency can strictly yield higher returns. While this might be theoretically straightforward, it underscores the practical reality that in certain tasks, reacting faster confers a distinct advantage. Indeed, when new or unmodeled information arrives between coarser time steps, lower-frequency controllers (or those that do not adapt quickly) risk missing events that incur large penalties or lose large rewards.

Concretely, we focus on safety-critical tasks where immediate reaction to collisions or unexpected contacts is crucial. In these tasks, an undetected collision or a delayed response can rapidly cause catastrophic damage. The “Safety Critical Reacher” environment (Section 6.2 of our paper) exemplifies this scenario: when there is a sudden collision, the policy must quickly backtrack, or else it will suffer a considerable penalty for the extra collision force. A lower-frequency controller often cannot respond fast enough within that sub-interval.

To further verify the existence of such MDPs, we introduce a new “Safety Critical Halfcheetah” environment with six degrees of freedom and an observation dimension of 19, exhibiting significantly more complex dynamics than our previous 1D safety task. Specifically, we add a blocking wall in 25% of the episodes, with collision force only observable upon contact. This wall blocks the cheetah’s forward path and incurs a penalty for any impact. As a result, the policy must quickly retreat if it detects a collision; otherwise, it continues forward. This design underscores the importance of higher-frequency actions for fast responses in unmodeled safety-critical situations, following a similar setup to "Safety Critical Reacher". Our full results (at https://imgur.com/a/LNBypMy) show that EvoControl still outperforms the baselines in this more challenging safety-critical scenario.

Table 33, Safety Critical Halfcheetah Results

(B) Why These Scenarios Are Important

These environments mirror real robotic applications, such as:

• High-gain robotic arms near humans: Where collisions must be minimized or mitigated the instant they occur, e.g., in collaborative assembly lines.
• Aerial vehicles under sudden gusts: Hovering drones or helicopters must correct torque and thrust when wind forces change abruptly.
• Surgical robotics: In certain procedures, micrometer-level slip or unintentional contact can cause tissue damage; responding at tens-of-milliseconds scale is vital.

In all these cases, the penalty for a delayed response can be high; hence, faster control loops can yield higher returns.

Furthermore, we validated the real-world feasibility of EvoControl on a 7-DoF Franka Emika Panda (Appendix J.14, p. 51), initially demonstrating superior collision handling and reduced contact forces compared to a tuned PD controller. Importantly, the policies trained in MuJoCo XLA transferred directly to the robot without requiring additional task-specific tuning, unlike fixed-PD controllers. These results underscore the practicality of using a learned fast controller in real-world scenarios.

proof/claim here seems superficial ... nothing wrong with keeping it.

While the proof (Proposition 2.1) is simple, we keep it to clarify that high-frequency control is not just an intuitive guess but can be rigorously shown beneficial under specific conditions.

not very novel (I would say the main novelty is incremental) Though high-/low-frequency control is common, our approach:

1. Learns a neural fast controller (vs. fixed PD/sub-goal tracking only policies).
2. Combines on-policy PPO for the high-level and ES for the low-level, learning jointly, avoiding instability from direct high-frequency RL.
3. Demonstrates consistent real-world performance with minimal tuning.

This synergy extends hierarchical RL to truly fast torque control while retaining exploration benefits from slower layers.

---

We hope that all of the reviewers’ concerns have been addressed. We’d be happy to engage in further discussions.

We thank the reviewer for their thoughtful and constructive feedback, particularly their appreciation that all the claims are well supported and the paper does a tremendous job supporting the findings, with the idea being novel and interesting. Below, we address each concern:

fixing the total time and not the number of steps is more fair. If possible, including the results for this will be beneficial.

This is an excellent point, we agree. We already provide results for all baselines in Appenidx J.4 (also shown here: https://imgur.com/a/CBzbEBO) where we fix the total time—here implemented by fixing the total number of low-level policy steps, which for all our high-frequency environments, that take actions at 500Hz, is a duration of 2 seconds. This ensures that the direct torque controller is evaluated in an identical physical time window to EvoControl. These new results still show EvoControl outperforming baselines, including the high-frequency direct-torque policy, which continues to struggle with long-horizon credit assignment at 500 Hz.

We will highlight these findings more prominently in the main paper’s Table 3 (as suggested) for camera-ready. This way, readers see both “fixed high-level steps” and “fixed total environment time” versions of the experiments.

Typos

Thank you, we have now fixed these.

Are there any experiments to show the proposed method has advantages over hierarchical RL methods with goal-reaching low-level policies, such as HIRO.

Yes, as HIRO closely resembles our "fixed-controllers" methods class (L192), where the low-level policies perform sub-goal state completion. Let us clarify that HIRO is an off-policy method that assigns state-based goals to a low-level policy, rewarded by sub-goal completion. However, as shown in Table 4, sub-goal attainment methods cannot handle the high-frequency reactions needed in safety-critical tasks.

We did use an existing implementation of HIRO (from the authors) on our high-frequency environments, however, we found that it was too unstable, and did not converge to a meaningful policy (showing random performance). We likely believe that HIRO, being an off-policy method, may require extensive hyper-parameter tuning for each environment and task to work correctly, and the authors within the HIRO paper discuss the instability of the method during learning.

Similarly, the Option-Critic framework uses a finite set of discrete options, each with learnable termination conditions. Because our tasks feature a continuous latent action (e.g., specifying a target joint position or velocity) and because we run on-policy PPO for the high-level layer, we found it non-trivial to combine discrete option-termination logic with continuous-time feedback.

Have you tried the techniques that enable stable training of other hierarchical RL algorithms such as option-critic and HIRO to stabilize the training of EvoControl instead of the annealing?

Thank you for the insightful question. The techniques used to stabilize other hierarchical RL algorithms do not directly apply to EvoControl as:

• HIRO is an off-policy method that relies on importance sampling and sub-goal relabeling of previously collected trajectories. By contrast, EvoControl’s high-level policy uses on-policy PPO, which discards old data once the policy updates. Hence, off-policy relabeling and importance sampling are not feasible in EvoControl.
• Option-critic typically defines a finite set of discrete options, each with its own policy and termination condition. Its stability arises from learning when to switch or terminate these options. In EvoControl, the lower-level controller is a single, continuous neural policy (e.g., torque commands at 500 Hz)—there are no discrete “skills” to terminate, and the entire low-level training is through ES on raw episodic return.

Consequently, we rely on annealing from a PD controller to the learned neural policy, which stabilizes our on-policy training and avoids the need for subgoal relabeling or discrete option terminations.

For example, different learning rates for the two levels to induce almost fixed behavior of one level when the other one is adjusting to its changes. I wonder why RL has failed to train the low-level policy here (Appendix J2) as opposed to those methods.

That is a nice idea. We empirically tested a similar concept, where, when using RL to train the low-level policy, we take interleaved updates every so many steps, and also did the same with EvoControl. Interestingly taking frequent interleaved updates causes instability, and taking interleaved updates slightly less frequently improves learning (e.g., take an update of the higher-level policy for every 16 low-level policy updates), this could arise due to an update on one level has a larger effect on the overall policy than an update on the other layer.

---

We hope that all of the reviewers’ concerns have been addressed. We’d be happy to engage in further discussions.