Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning

We are extremely grateful for the insightful review, feedback, and opportunity to improve our work! We are glad you found the issue we address important, our solution straightforward and broadly applicable, and our results thorough and well-documented. We address all concerns and questions below:

Weaknesses:

I am concerned about the originality of the proposed method.

We believe the primary contribution of this work is the systematic analysis, characterization, and empirical justification of this technique. We introduce and characterize the problem setting, empirically demonstrate its prevalence in modern RL training regimes through empirics, identify which RL environments can maximally benefit from staggered resets through the toy experiments, and provide practical implementation details such as balancing the tradeoff between environment reset overhead and rollout uniformity. By doing so, we hope to provide others in the field with a best practice implementation step that can improve performance on long-horizon tasks in the massively parallel (and popular) RL setting. We note that many priors works published at past top conferences are similarly motivated; their value is derived from investigating the "nuts and bolts" of RL algorithms since small implementation details can massively impact training performance [A, B, C].

While the method’s simplicity is a strength, I am concerned about the use of a deterministic sampling scheme. It's not hard to imagine environments where this determinism could be exploited or lead to failure.

Our choice of a deterministic grid-based staggering approach was a deliberate design decision to balance theoretical purity with computational efficiency and learning stability. On modern GPU environments (IsaacGym, ManiSkill, etc), calling env.reset() in serial incurs computational overhead, so there is a fundamental tradeoff between how deterministic / granular the staggering scheme is versus how parallelized environment resets are. We empirically validate this design choice in an experiment featured in the appendix (starting L648), where we find that, in real-world wall-clock performance, the deterministic scheme with fewer reset gates outperforms the case in which there is a reset gate at every step, which is provably the same as a randomized staggering scheme.

Questions:

Line 163: It's unclear why you specify values for $H$ and $K$ here.

This is an error in the manuscript, as the claim indeed holds whenever $H >> K$. We will correct the future version of the manuscript to remove this specification.

I'm a bit confused by the design of your toy environment. Since the states evolve independently of the agent's actions, it seems more appropriate to describe this setup as a bandit problem rather than an MDP—is that accurate?

While the toy environments are somewhat inspired by bandits, the setting is different as states depend on the agent’s actions whenever $p_{prog} < 1$. For the majority of the toy experiments, $p_{prog} < 1$ (except for when we directly vary it to measure the effect of skill gating, and then for one experiment $p_{prog} = 1$), and hence we are not operating in the bandit setting for the vast majority of toy experiments.

But isn't the agent always in the final block at the end due to the deterministic structure of time steps? Additionally, the notion of "success" in this context is not clearly defined—could you clarify what it means?

You are correct that, when $p_{prog} = 1$, success as we define it is guaranteed. We found return to be a far more illustrative metric, and hence all of our empirical analysis of the toy environments focuses on return rather than success. We appreciate your observation and will modify the future manuscript, as the inclusion of success criteria is unnecessary and understandably confusing to the reader.

In line 204, I also disagree with the statement that $b_0$ is "centered around 0" - it's centered around $\lambda_R$

This is an error in the manuscript. We will promptly update the manuscript to reflect this.

Why do you only consider $\lambda_R \in [0, 2]?$

We vary $\lambda_R$ from 0 to 2 because, as shown in Figure 2b, the difference between naive synchronous rollouts and staggered resets shrinks and the intended effect saturates. To provide some empirics, we ran the experiment again with $\lambda_R = 10$ for 3 seeds and found that staggered resets achieves final return of $18.148 \pm 0.149$ and naive achieves $17.998 \pm 0.223$, confirming your expectation that the difference vanishes as $\lambda_R$ increases. In the final manuscript, we will increase the range of values of $\lambda_R$ in Figure 2b to better demonstrate this effect.

Line 303: You mention that "locomotion environments feature highly stochastic dynamics." Could you elaborate on this? Aren't locomotion tasks essentially deterministic?

Our claim referred to the overall training scheme which deliberately introduces significant domain randomization into the RL environment for sim-to-real transfer [D]. Furthermore, as we claim, a crucial source of randomness comes from the reset behavior itself, as robots attempting locomotion fall at different, unpredictable times. The combination of domain randomization and asynchronous failures introduces natural desynchronization, explaining (inline with the analysis from our toy experiments section) why the marginal benefit of our method is less pronounced for pure locomotion tasks. We will revise our claim in the future manuscript to be more precise.

Is the number of blocks fixed at $H / K$ across all values of $N$ in these plots?

We fix the number of blocks to be $H / L$, where $H$ is the maximum horizon and $L$ is the block length. This is detailed in the appendix but we will revise the main text to be more clear here.

Did you consider formulating the problem using a discounted infinite-horizon setting, where each block has an episode length $H$ drawn from a geometric distribution with parameter $1 - \gamma'$ for some $\gamma' > \gamma$ (and resampled for each block after termination)?

This is an alternate way to vary the level of stochasticity in environment dynamics. Instead of varying the stochasticity of the reset distribution, your idea involves varying the stochasticity of inter-block dynamics, prior to reset. In the toy setting, we could imagine both approaches yielding similar results, except for the behavior of early resets. We believe this phenomenon of frequent partial resets relates to our findings on the high-dimensional robotics tasks. Locomotion, which experiences frequent partial resets due to early failures or successes, benefits less from staggering resets compared to other long-horizon robotics tasks, which only reset early when they succeed, which requires the policy to have almost fully converged. Hence, the natural uniformity in rollout buffer that can be achieved by frequent early resets usually does not occur in practice, because early resets generally require the training to already be near / at completion.

An alternative way to address the coverage issue you're targeting could be to sample initial states from a more diverse distribution (without staggering environments). Could you elaborate on whether there are any theoretical differences between this alternative and your approach?

We thought about whether a modified reset distribution could, in theory, afford the same benefits as staggering. consider a task with $H = 100$, 100 parallel environments, and 100 states. Scenario 1: Let the standard reset distribution $D$ just be state 1. Say we applied staggered resets with 100 reset gates. Scenario 2: Let the modified reset distribution $D’$ be the uniform distribution across all 100 states. One could imagine, under certain stationarity assumptions of the environment’s state space, these two scenarios to be provably equivalent. However, we find that, despite this interesting theoretical connection, staggered resets is more practical than solutions focusing on the states that an environment resets to because obtaining a suitable $D'$ is highly nontrivial. This is because, for high-dimensional tasks, resetting is highly nontrivial and current environment resetting routines in GPU simulations involve far more than sampling random vectors. Additionally, for tasks in which the majority of the set of reachable states is not known, staggered resets afford rollout uniformity whereas $D’$ would have to change with each rollout to incorporate newly visited states. Because, for high-dimensional RL tasks, the set of all reachable states is a superset of the support of the reset distribution, we would constantly need to add to an ever-growing support of states in our reset distribution. This requires far more overhead from an engineering/implementation perspective. In general, we (empirically) and the field at large find that more diverse reset distributions yield better performance [D]. However, as previously mentioned, this is highly nontrivial to achieve performantly at scale, and hence staggered resets are highly appealing as a lightweight drop-in solution that can be applied to any environment, regardless of the semantics of its reset dynamics.

We would like to once again thank you for your exceptionally thorough and constructive review. We hope our response address your concerns. In light of these clarifications, would you be able to raise your score? We are happy to address any more issues you have.

References:

[A] FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control, Younggyo Seo et al. 2025

[B] Implementation Matters in Deep RL: A Case Study on PPO and TRPO, Logan Engstrom et al. ICLR 2020

[C] Efficient Online Reinforcement Learning with Offline Data, Philip J. Ball et al. ICML 2023

[D] Learning to Walk in Minutes using Massively Parallel Deep Reinforcement Learning. Nikita Rudin et al. CoRL 2021

We are extremely grateful for the insightful review, feedback, and opportunity to improve our work! We are glad you found the approach straightforward, well-explained, and our experiments well thought out. We address all concerns and questions below:

Weaknesses:

the results on the high dimensional robotic control tasks (for which this method is supposed to be maximally beneficial) to be promising, but quite limited.

We appreciate your concern about results on high-dimensional robotic control tasks being limited. We emphasize that, to our knowledge, this is the primary domain of interest in which the massively parallel on-policy RL setting is applied. Hence, we reasoned that this domain is the ideal testbed for our method’s intended application, as well as a primary area of interest for practitioners interested in applying our work. However, we are working on follow-up experiments on game environments such as Atari and Pong, which demonstrate similar long-horizon tendencies and for which there exist vectorised environments for. We hope to include these results in a future version of the manuscript for additional empirical justification of our method.

Only 3 random seeds makes it difficult to draw strong conclusions ... The StackCube and PushT tasks seem to indicate greater sample efficiency but less training stability as the performance degrades over time. To me this seems contradictory to a core point the authors are making which is that staggered resets improves training stability. Further along this point there appears to be high variance for the training performance on TwoRobotPickCube for staggered resets.

We appreciate the concern about the small number of random seeds used in the evaluation of this method. We agree with you, and hence we are running experiments with 10 seeds rather than 3. We started with StackCube-v1 and TwoRobotPickCube-v1 to address your subsequent concerns about training stability and high variance between training runs. Our findings indicate that the proposed method is actually far better at improving training stability than the original empirics would indicate. For StackCube-v1, we have environment evaluation reward at 0.71 +/- 0.026 with staggered resets and 0.63 +/- 0.04 without staggered resets (n = 10), after training for approximately 25M environment steps. For the same experiments, we also have success rate at 0.80 +/- 0.04 with staggered resets and 0.31 +/- 0.41 without staggered resets. For TwoRobotPickCube-v1, we have environment evaluation reward at 0.77 +/- 0.006 with staggered resets and 0.40 +/- 0.171 without staggered resets (n = 10), after training for approximately 50M environment steps. For the same experiments, we also have success rate at 0.949 +/- 0.013 with staggered resets and 0.027 +/- 0.13 without staggered resets. In both cases, using staggered resets provides clear advantages both in terms of increased average evaluation reward and evaluation success rate, but also in reduced variance of these quantities compared to using naive synchronous rollouts, further supporting our original claim that staggered resets reduces the variance across seeds. We firmly believe these additional results strengthen the statistical significance of our results, addressing one of your foremost concerns with the paper in its current state. We plan to have the remaining empirics added as soon as possible. We also have a table with these results, since we can't share the return/success curves directly as per NeurIPS rebuttal policy:

Questions:

In Figure 3 systems are trained for the same amount of environment steps for all tasks except on UnitreeG1Transport-Box, is this because staggered resets also suffered from policy collapse there?

We wanted to extend the curve for the UnitreeG1Transport-Box-v1 task since PPO with naive synchronous resets would otherwise appear to have never learned a successful policy, when in reality it simply learns far slower than when equipped with staggered resets. This explains the difference in the number of environment steps.

Have the authors considered the case where environments were only reset synchronously once and then allowed to automatically terminate and reset without having to wait for any reset gates?

Allowing for early resets comes with a myriad of considerations and tradeoffs. On one hand, it allows for better utilization of the vectorized environment, as an environment that has terminated early can begin collecting useful data quicker. However, calling env.reset() when not in-sync with reset gates will incur additional overhead if the environment’s reset routine is involved, as discussed in the wall-time analysis section of the appendix. Additionally, there is some risk in disrupting the uniformity of the rollout buffer afforded by staggered resets. Ultimately, we didn’t observe significant empirical differences in wall-clock convergence time when enabling versus disabling early resets in the high-dimensional robotics tasks. We are happy to include more information on this tradeoff in the final manuscript.

How were the hyperparameters for high dimensional robotic control benchmarks chosen?

We simply choose the same hyperparameters as used in the ManiSkill3 baseline implementations. The chosen values are also commonly used in other works involving high-dimensional robotics tasks in the massively parallel training regime, such as SAPG [A].

The 1D toy ablation environment is defined as having B blocks (or levels) when initialised, but I cannot find a value for B in the text. Can the authors please mention what this value is?

The value of $B$ is dependent on the horizon length $(H)$ and block length $(L)$. We fixed $L$ as 5 steps across all experiments. For instance, a 200-step horizon toy environment would yield $B = 40$ blocks/levels in the environment. We will update the paper to better clarify this.

The method relies on using some heuristic like no-op actions or random actions to progress environments to start at varying locations in time. Is there any impact on training based on the method used to do this?

Staggered resets can be implemented either by (1) advancing certain environments using heuristic actions or (2) modifying the elapsed steps attribute of the vectorized environment itself. We found that, experimentally, there is no difference in performance or training based on either method, and that the choice between the two mainly depends on engineering/implementation semantics. For instance, some vectorized environments may not provide mutable access to the vector storing the number of elapsed steps for each environment. Similarly, some vectorized environments may not easily support only resetting or stepping a certain subset of the total environments. We appreciate the thoughtful question, and will add a section on these implementation considerations to the final manuscript.

What if an environment can not be suitably advanced in time with a heuristic?

We appreciate your thoughtful question. We tackle this question in our skill gating toy experiment (Figure 2c). In that experiment, when the progression probability is low, the agent cannot advance to later stages without first mastering earlier skills. Our results demonstrate that the advantage of staggered resets diminishes as these skill gates become stricter, as the environment itself imposes a curriculum that naturally forces the agent to spend more time on early-episode states, reducing the impact of cyclical nonstationarity. However, we note that, even with $p_{prog} = 0$ (the strictest case of skill gating), the agent initialized with staggered resets achieves a return of 60, compared to 20 without staggered resets, a ~3x increase in reward. Our explanation for staggered resets’ better performance even with strong skill gating is that, despite the states not being advanced when the heuristic action is performed at initialization, the long-term reset behavior is changed for the length of the training run, and hence when the effective skill horizon increases as training progresses, staggered resets begin to benefit the agent when cyclical nonstationarity may start to emerge. We hope this clarifies your question, and we are happy to elaborate further and incorporate elements of this explanation into the revised manuscript.

What is the rollout length K that is kept fixed as the episode horizon H is increased in Figure 2a?

Rollouts are 5 steps long for the experiments shown in Figure 2a. This detail $(K = 5)$ is included in the appendix, line 518, and we can revise the main manuscript to make this clearer.

We would like to once again thank you for your exceptionally thorough and constructive review. We hope our response address your concerns. In light of these clarifications, would you be able to raise your score? We are happy to address any more issues you have.

[A] SAPG: Split and Aggregate Policy Gradients, Jayesh Singla et al. ICML, 2024.

Thank you for the follow-up and opportunity to clarify your question.

On UnitreeG1Transport-Box and Early Stopping:

Our PPO implementation included an early stopping feature, where if an agent achieves an evaluation success rate past a certain threshold (0.975 in our implementation), the training run automatically terminates. On UnitreeG1Transport-Box, this happens for every seed we originally tested. However, we ran new experiments for a few seeds with early stopping disabled and found no evidence of policy collapse with staggered resets. Rather, the agent maintains near-constant reward and success rate after convergence. We cannot share the exact return curves as per NeurIPS rebuttal policy, so we share success rate at various points in training for these new results. We can clarify this point in the updated manuscript, as we agree it can be confusing to readers.

On Potential Policy Collapse in StackCube and PushT:

Regarding your comment on policy collapse in StackCube and PushT, at the request of you and other reviewers, we ran far more seeds ($n = 10$ instead of $n = 3$) to strengthen the statistical significance of our results. For StackCube, any sign of policy collapse in aggregate metrics (reward/return/success rate) disappears with $n = 10$. We provide the discretized success rate curve below:

For PushT, your comment led us to re-examine our evaluation criteria. We found that our initial metric for this environment measured the number of successes at the final timestep of the episode. However, we realize this is a flawed success metric, since as the agent becomes more proficient, it often solve the task earlier in the episode. Since reward is not shaped adequately to maintain this success state for many steps, subsequent actions can inadvertently undo it by the final timestep, causing this particular metric to decrease. We implemented a more appropriate metric: success as registered by whether the success criteria is observed at any point in the episode. With this revised metric, success rate remains high after convergence for the agent initialized with staggered resets. We provide the discretized success rate curve below:

Hence, we claim that policy collapse is not a concern in this work, and we will update our presentation with the greater number of seeds to properly convey this. We appreciate your follow-up question and we hope this addresses any remaining concerns you may have.

I would like to thank the authors for taking the time to answer my questions. I will maintain my current score, in line with Reviewer jQjk's recommendation.

The number of seeds in the experiments is insufficient.

We are running experiments with 10 seeds rather than 3. We started with StackCube-v1 and TwoRobotPickCube-v1 to address your subsequent concerns about training stability and high variance between training runs. Our findings indicate that the proposed method is actually far better at improving training stability than the original empirics would indicate. For StackCube-v1, we have environment evaluation reward at 0.71 +/- 0.026 with staggered resets and 0.63 +/- 0.04 without staggered resets (n = 10), after training for approximately 25M environment steps. For the same experiments, we also have success rate at 0.80 +/- 0.04 with staggered resets and 0.31 +/- 0.41 without staggered resets. For TwoRobotPickCube-v1, we have environment evaluation reward at 0.77 +/- 0.006 with staggered resets and 0.40 +/- 0.171 without staggered resets (n = 10), after training for approximately 50M environment steps. For the same experiments, we also have success rate at 0.949 +/- 0.013 with staggered resets and 0.027 +/- 0.13 without staggered resets. In both cases, using staggered resets provides clear advantages both in terms of increased average evaluation reward and evaluation success rate, but also in reduced variance of these quantities compared to using naive synchronous rollouts, further supporting our original claim that staggered resets reduces the variance across seeds. We firmly believe these additional results strengthen the statistical significance of our results, addressing one of your foremost concerns with the paper in its current state. We plan to have the remaining empirics added as soon as possible.

Questions:

Can you elaborate on the structure of the toy MDP?

You are correct -- when $p_{prog} = 1$, the agent advances through the level after the specified number of steps. However, return is dependent on the agent's accuracy at each level, and hence the task is still nontrivial when $p_{prog} = 1$. We will modify our manuscript to better explain this case.

We hope our response address your concerns. In light of these clarifications, would you be able to raise your score? We are happy to address any more issues you have.

References:

[A] Learning to Walk in Minutes using Massively Parallel Deep Reinforcement Learning. Nikita Rudin et al. CoRL 2021

[B] SAPG: Split and Aggregate Policy Gradients, Jayesh Singla et al. ICML, 2024.

[C] ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI, Stone Tao et al. RSS 2025.

[D] Visuomotor Policies to Grasp Anything with Dexterous Hands, Ritvik Singh et al. 2024

[E] Extreme Parkour with Legged Robots, Xuxin Cheng et al. ICRA 2024

Second, we quantify memory retention with a “forgetting matrix”--the per-state/level drop in accuracy relative to the best historical accuracy on that particular state/level. In the naive setting (no staggering), accuracy drops on average by 0.21, dropping maximally by 0.6 over the course of training. With staggered resets, accuracy only drops on average by 0.015, never exceeding 0.05. Thus, the agent that receives staggered resets retains knowledge of earlier blocks ~14x better, providing empirical evidence towards the claim we make about information forgetting.

In conclusion, we have strong empirical evidence that the cyclical shift in input distribution can destabilize value function learning and hinder the agent’s ability to consolidate knowledge across the entire task horizon.

the paper argues that the proposed approach is algorithm-agnostic. While on paper that may be true, whether staggered resets bring benefits to other algorithms has not been studied...

We are running experiments with SAPG, a newer on-policy RL algorithm. Initial results indicate staggered resets provide a similar benefit to SAPG as they do to PPO (as SAPG operates in the same $K << H$ regime) but we are waiting on more seeds to confirm.

... the actual locomotion skill itself is much shorter horizon.” This is a conjecture that is not supported by evidence.

This claim captures an intuition / observation, and we agree empirical evidence should support it. We ran the same KDE experiment for locomotion environments (specifically MSHumanoidWalk-v1, and ideally more soon), and despite long horizons (up to $H = 1000$), task length as judged by cycle period is far shorter. Due to NeurIPS rebuttal policy, we cannot share these figures unfortunately, but we will include these observations in the final manuscript.

The text claims that the problem of cyclic non-stationarity is a general problem for on-policy RL in the introduction but only results on one algorithm (i.e. PPO) are presented.

As previously mentioned, we are running follow-up experiments on other on-policy RL algorithms.

Relation to Scientific Literature:

We will include the necessary citations as requested for these claims we make. As you mention, nonstationarity has played a large role in RL, and hence we will update the related works section along with the citations for the specific claims we make. To our knowledge, no works have examined this particular problem of nonstationarity induced by the $K << H$ regime. Finally, for the claim about partial resets, many prior works cover this for both manipulation and locomotion tasks, which we will accordingly cite in our revised manuscript [A] [C] [D] [E]

Experimental Design and Analyses:

Given that tasks are only chosen from a single benchmark, the experimental section lacks variety.

To our knowledge, this is the primary domain of interest in which the massively parallel on-policy RL setting is applied. Hence, we reasoned that this domain is the ideal testbed for our method’s intended application, as well as a primary area of interest for practitioners interested in applying our work. Prior works such as SAPG also only focus on robotics. However, we ran follow-up experiments on MinAtar environments. On Breakout-MinAtar, we train for 4e6 environment steps with rollout lengths of 8, with 1024 parallel environments. Staggered resets achieves episode return of $59 \pm 6.5$, compared to naive rollouts achieving $31 \pm 11.2$ for 7 seeds. Once again, we are unable to share the return curves directly as per NeurIPS policy, but these results provide variety and strengthen our claims, and we hope to incorporate more such empirics from other environments to the final paper.

(continued in next comment)

We are extremely grateful for the insightful review, feedback, and opportunity to improve our work! We are glad you found the problem we study interesting and important to the field. We address all concerns and questions below:

The problem itself is not well established in the text. Whether or not short rollouts are in fact needed is unclear and is only evidenced by one study in the paper

We appreciate your concern and candidness on this point. To the best of our knowledge, empirical work on choosing $K << H$ as optimal hyperparameters has not been commented on in prior literature, besides in [A]. [A] claim that they decrease the rollout length as they scale up the number of parallel environments to maintain a reasonable batch size. Additionally, if we examine more recent works on scaling massively parallel on-policy RL, such as SAPG [B], we find that they also opt for rollouts of between 8 and 16 steps, even for environments with horizons exceeding 50 steps, such as Shadow Hand and AllegroKuka manipulation tasks. Despite this additional evidence from the literature, we ran a sweep of new experiments to provide additional empirical evidence as to the relevance of this problem setting. For StackCube-v1, which has a task horizon of 100 steps, we fix optimal hyperparameters and vary the rollout length in [1, 2, 4, 8, 16, 32, 64, 100]. We run these experiments for 3 seeds (currently running more), with both staggered resets enabled and disabled.

Table 1: Final reward after 100M environment steps for StackCube-v1

Table 2: Environment steps (in millions) to reach >75% success rate

DNC (Does Not Converge) indicates the agent did not meet the threshold within 100M steps

Table 3: Wall-clock time (in minutes) to reach >75% success rate

Note that the fastest wall-clock times are achieved with shorter rollouts (K=8 and K=16) on an NVIDIA RTX 4090

These new empirics shine light on why practitioners are drawn to the $K << H$ regime. Note that, in table 1, final reward saturates at 8-step rollouts, both with and without staggered resets. Additionally, as shown in Table 2 and 3, training runs with shorter rollouts (and hence higher update-to-data ratios) achieve significantly faster wall-clock convergence times and require less net environment steps. Hence, as evidenced by multiple past works and our own new set of experiments, we firmly believe that this problem setting is well-motivated. We will accordingly update the manuscript.

Claims and Evidence:

At no point does the manuscript measure value functions, policy oscillations or a form of forgetting/information gain ... There is no measure of information consolidation or forgetting that is being reported. As mentioned before, more metrics would benefit the paper ... This cyclical bias prevents the learner from accurately estimating values and advantages, likely due to catastrophic forgetting phenomena” - No evidence is presented for this (L36, L220, L135)

We ran a revised set of experiments across 3 seeds on the toy environments to address these concerns. During training, we log two metrics. First, we quantify value function stability by measuring the mean-squared prediction error between the critic’s prediction V_{\theta)(s) and the Monte-Carlo / GAE target used for that minibatch. In the naive setting (no staggered resets), the value function estimation error catastrophically surges to around 20-100 MSE at three points in training – at updates 40/80/120, where all environments reset together. Comparatively, with staggering, the value function estimation error never exceeds 2 MSE, and does not experience catastrophic surges at certain updates. These data provide evidence that the cyclical bias hinders the learner from accurately and stably estimating values.

(continued in next comment)

Dear authors,

thank you for the very detailed rebuttal and all the additional effort you are putting in based on my feedback. I believe the inclusion of the motivational example, additional metrics and additional algorithms will make this manuscript significantly stronger.

That being said, while I appreciate the effort towards statistical rigor unfortunately I cannot recommend acceptance of a scientific manuscript whose results are not yet finished and cannot be reviewed. I will raise my score to 3 to indicate that this paper would likely have been an accept recommendation from me if all the results were in and conclusive.

Side note: for additional environments, I agree that robotics is an interesting application specific benchmarks reward functions have specific peculiarities. It might make sense to consider looking into MuJoCo Playground for task variety.

Dear Reviewer,

We are glad to hear that our additions will significantly strengthen the manuscript. To address your final claim about the unfinished empirics, we have provided additional empirics as requested:

We ran all ManiSkill experiments included in the original paper for 10 seeds rather than 3. In addition to the two environments' results we provided earlier, we provide results for all six environments below:

StackCube-v1 $(n = 10)$

UnitreeG1Transport-Box $(n = 10)$

AnymalC-Reach $(n = 10)$

PushT $(n = 10)$

TwoRobotPickCube $(n = 10)$

MS-HumanoidWalk $(n = 10)$

We have also tested the AllegroKuka manipulation tasks that are used in the SAPG environments with the SAPG algorithm, to further provide evidence to our claim that our technique is algorithm-agnostic. We found that the SAPG codebase has many conditions for early resets, such as the object falling, success, and the hand being far from the object. Additionally, we find significant domain randomization in the reset dynamics of the environments. In line with our earlier findings on the ManiSkill environments and toy experiments, we find that these environment features reduce the cyclicity problem. However, these early resets reduce throughput of the GPU environment considerable. Regardless, staggered resets outperform naive SAPG with or without early resets enabled. We present results on two AllegroKuka tasks with early resets enabled below:

Allegro Kuka Regrasping $(n = 5)$

Allegro Kuka Throw $(n = 5)$

We once again thank you for the pertinent and detailed feedback, and hope that these additional empirics are conclusive and establish the level of statistical rigor that you ask for. In light of these new empirics, would you be able to raise your score? We once again thank you for your time spent reviewing our manuscript — our conversations have been productive and we have learned a lot.

We are extremely grateful for the insightful review, feedback, and opportunity to improve our work! We are glad you found our approach simple and effective, and we address your questions/concerns below:

Presentation of the method can be improved. Section 3.3 introduces the method with a large bulk of text, but it would be easier to follow if a pseudo-code is provided.

We will modify this section of the manuscript to be more approachable and digestible to readers.

The proposed method is positioned as a method to improve the performance of massively parallel PPO training. If so, a comparison with other works that aim to improve PPO performance in large-scale parallel training is needed.

We agree that SAPG is another similarly motivated work focused on improving the scaling performance of massively parallel on-policy RL, albeit requiring several heavily involved modifications to the core PPO algorithm. We have begun conducting follow-up experiments with the SAPG codebase/environments in order to provide ample comparison. However, we also wish to emphasize that our work is complementary to works such as SAPG, rather than standing as a point of comparison. Initial results indicate that staggered resets have a similar effect with SAPG as the policy gradient oracle as they do with PPO, but we are waiting on more seeds/environments before conclusively claiming this. We will update with more empirics shortly, and appreciate the concern about the lack of variety in experiments.

In addition, other environment reset strategies may be considered. For instance, one can uniformly sample an initial state from the environment or from the history of the agent. I believe these strategies could have a similar effect, too, and are easy to implement.

We have considered similar strategies during the research process, and have discarded them for various reasons. Sampling a vector from the state space of the environment uniformly does not perform well in practice, as the manifold of reachable states is far smaller than the total space spanned by $\mathbb{R}^D$ in practice, for a $D$-dimensional state. Hence, large amounts of rollout data are wasted by initializations to unreachable and hence useless initial states. I agree that, under certain assumptions, your second idea could have a similar effect to staggered resets, but we found that the implementation overhead to be too impractical in practice. In regimes where the size of the set of reachable states exceeds the size of the support of the environment reset distribution, the size of the replay buffer required to mimic the effect of staggered resets grows too fast. Additionally, as developing good representations of high-dimensional RL environment states is an open and ongoing challenge, we found it difficult to compress these replay buffers to the extent to which any performance benefits from uniformity gains exceed the increase in overhead. Both ideas empirically underperformed the PPO baseline in our initial experiments, and hence we chose not to pursue them further / include them in our high-dimensional robotics results. We appreciate your comment on this, and will update the manuscript with a description of our reasoning / findings on these other environment reset strategies.

Secondly, the number of parallel environments studied is relatively small compared with SAPG that use 20K environments.

With our follow-up experiments with the SAPG codebase, we aim to increase the number of parallel environments from ~6-10k (ManiSkill). However, we note that the level of parallelization required to run 20k environments in parallel necessitates multi-GPU training, which entails an entirely new set of engineering and implementation considerations regarding throughput and communication bandwidth. Hence, we will only scale up the number of parallel environment as far as a single GPU can support. While the better utilization of parallel environments is a secondary benefit of this technique, its primary purpose is to mitigate the cyclical nonstationarity problem we observe.

Questions:

What does each row and column mean? From the caption, it seems that (a) represents a long rollout but the bottom of the figure shows rollout 1, 2, 3 .... I'm not sure if I'm following the figure.

We agree that this aspect of the figure is confusing. (b) and (c) are 25-step rollouts but (a) is a 100-step rollout. We will modify this figure in the final manuscript to rectify this error, and we appreciate your observation.

We hope our response addresses your concerns, and we are happy to address any more issues you have. We are once again extremely grateful for this opportunity to improve our work, and appreciate the effort you put into your review.