Suggestions

"In the experiments that the authors conducted, it seems the proposed methods do not show better performance than baseline PPO, especially since it is unclear which implementation version of baseline PPO the authors used..."

We respectfully disagree that the proposed methods do not show better performance over baseline PPO. Figures 3, 4, 8, and 9 show a robust, statistically significant improvement across three metrics (median, IQM, mean) over an aggressively tuned per-task PPO baseline. We understand there may be confusion with the hyperparameter sensitivity plots, where common outer-PPO hyperparameters are shared across a suite to represent normalized performance across the grid range. We have amended the captions to Figures 5, 6 and 7 to make this distinction more clear. We emphasize that the sensitivity plots share outer hyperparameters hence do not achieve the stronger performance of the task-specific hyperparameters used in Figures 3, 4, 8, and 9. However despite the lack of clear improvement given common hyperparameters, the sensitivity plots do show that learning is stable and performance is robust for a wide range of values, particularly on Brax and Jumanji. We lastly highlight that the inner PPO hyperparameters are themselves task-specific hence we would not expect common values to be optimal.

We thank the reviewer for highlighting the lack of clarity concerning the PPO implementation, and share their concern for the sensitivity of PPO to implementation details. As stated in line 813 we used the Stoix implementation of PPO, which is publicly available here (https://github.com/EdanToledo/Stoix). We have also provided our code implementation in the supplementary material. To further provide clarity on the implementation details we have provided an additional Table 1 in Appendix B in which we exhaustively define the implementation details employed (or omitted) in our work as identified by the works [3, 4]. We thank the reviewer for highlighting this, as we believe this addition will enhance the reproducibility and transparency of our results.

"It would make this paper stronger if the authors could provide the details of the training time of each algorithm with the same number of timesteps."

We have added Appendix H in which we discuss the computational complexity of the outer-PPO methods, and provide a Table 6 in which the runtimes are compared. As the outer-PPO methods do not materially increase the computational complexity, we observe no significant deviation in runtime between the different methods.

We deeply appreciate your time and effort in reviewing our work and providing valuable feedback. We hope that our response and amendments, particularly the addition of Appendices H, I and K, address your concerns and if so, we would be grateful if you could consider updating your score. We would of course of happy to respond to any remaining questions you may have.

[1] Deep Reinforcement Learning at the Edge of the Statistical Precipice [2] Discovered Policy Optimisation [3] Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO [4] The 37 Implementation Details of Proximal Policy Optimization

"No strong justification is given why we should expect the new method to lead to large improvements in performance"

We wish to draw the reviewers attention to the three research questions we highlight in lines 48 - 52. The contribution of this work is to (a) identify these implicit design choices exist, (b) propose a method than enables these design choices to be relaxed, and $c$ empirically validate that in all three cases the design choice is suboptimal. We believe the common understanding before our work would have been that all three design choices are necessary for optimal performance, particularly the outer learning rate of 1 and lack of outer loop momentum. Therefore, our results stand in contrast to the field's understanding of one of the most commonly used algorithms across reinforcement learning, which we believe empirically alone is a strong research contribution. We further provide intuition for how these methods are modulating the PPO update rule, and how these can be motivated in Sections 3.1, 3.2 and 3.3. We appreciate the reviewers comment that a stronger justification would be welcomed, but believe this is beyond the scope of this initial empirical investigation, which we hope will stimulate further research both empirical and theoretical to understand the properties of outer-PPO and outer-variants of other policy gradient methods.

"The outer learning rate sensitivity plots show that the peak performance is not much different compared to sigma=1, which corresponds to standard PPO"

We understand there may be confusion with the hyperparameter sensitivity plots, where common outer-PPO hyperparameters are shared across a suite to represent normalized performance across the grid range. We have amended the captions of Figures 5, 6 and 7 make this distinction more clear. We emphasize that the sensitivity plots share outer hyperparameters hence do not achieve the stronger performance of the task-specific hyperparameters used in Figures 3, 4, 8, and 9. However despite the lack of clear improvement given common hyperparameters, the sensitivity plots do show that learning is stable and performance is robust for a wide range of values, particularly on Brax and Jumanji. We lastly highlight that the inner PPO hyperparameters are themselves task-specific hence we would not expect common values to be optimal.

"For the outer-PPO hyperparameter tuning, did you also first tune using 4 seeds, and then run a final evaluation using 64 new seeds, just like for the standard PPO experiments?"

Yes, the grid search was performed using 4 seeds per trial with final evaluation using 64 new seeds. The same set of seeds was used for evaluation on all methods.

We are sincerely thankful for your thoughtful review and the time you have taken to engage with our work. Should our explanations and amendments sufficiently address your concerns, we kindly ask you to consider your score. We would be eager to engage further if you have any questions remaining.

Response to Questions:

“I ask the authors to clarify what is the advantage of introducing the learning rate instead of modifying $\epsilon$.”

The reviewer also provided further context on this question in the “Weaknesses” section of their review.

“Furthermore, I am unsure whether the learning rate is necessary: the hyperparameter of PPO already provides a mechanism to control "how aggressive" the policy updates are. While I can acknowledge that the learning rate and the $\epsilon$ are two different terms (i.e., the learning rage acts in the parameter space, while the hyperparameter acts in the "policy" space), I can't see what is the advantage of using $\sigma$ in place of modifying $\epsilon$.”

As the reviewer has noted, $\epsilon$ controls the size of the trust region in policy space, while the outer learning rate operates in parameter space, scaling the outer gradient vector directly. This decoupling allows outer-PPO to amplify or attenuate updates without altering the surrogate objective. We have added an appendix highlighting the new behavior introduced by outer-PPO, that cannot be recovered by standard PPO in Appendix G. We motivate these new behaviors in lines 202 - 215. Attenuating the update direction can be motivated as ‘not trusting’ any given outer gradient, for reasons such as the noise present in data collection and stochastic optimization, irrespective of the outer gradient magnitude (e.g using different values of $\epsilon$). Another motivation is the potential lack of monotonicity of improvement along the linear interpolation $\theta_k + \sigma(\theta_k^* - \theta_k)$ for $\sigma \in[0,1]$ arising from the non-linear map from parameters to policy and non-convex surrogate objective, which we have added in line 207. Amplifying the update vector with an outer learning rate $\sigma > 1$ can be motivated as encoding confidence in the update direction. This may be desirable to use on well-estimated low-$\epsilon$ outer gradients. We posit in the paper that optimizers in the outer loop (e.g. momentum) enhance performance in ways that modifying epsilon alone cannot achieve, as evidenced by the superior performance within a given hyperparameters tuning budget. We lastly draw the reviewers attention to the three research questions we seek to resolve in the introduction, and emphasize that we did not seek to find the highest performing outer-PPO configuration, but to answer the aforementioned questions regarding implicit design choices of standard PPO.

“Perhaps, as highlighted in the "Strength" section, the main weak point of the paper is the significance”

We view this work as a proof-of-concept exploration into the outer-loop optimization using PPO surrogate optimization as the inner-loop. The significance lies not only in the observed performance gains but also in challenging previously unquestioned assumptions in PPO design. For example, the success of outer learning rates $\sigma > 1$ is surprising given PPO's original motivation of conservative policy updates. This finding suggests opportunities for further innovation in dual-loop algorithms, as our approach is broadly applicable beyond PPO to other RL algorithms. We also underscore the rigor of our investigation, including robust baseline comparisons and comprehensive hyperparameter sensitivity analyses. This methodological contribution supports future research in outer-loop methods for reinforcement learning.

Thank you for your thorough review and the effort you have invested in helping us improve our work. We hope our comments and the addition of Appendix G address your concerns and if so, would greatly appreciate your consideration in revising your score. We are of course eager to answer any further questions you may have.

Response to Questions

1. We have used "exactly coupled" to emphasize a property of standard PPO, where the behavior parameters at each iteration are exactly the solution to the previous inner-loop surrogate objective, without any additional transformations. While the term "behavior parameters" is not new in reinforcement learning (referring to the parameters defining the data-collecting policy), we agree that emphasizing this term would improve clarity, and have amended this line to do so.

2. The intuition behind biased initialization is to leverage prior trajectory information to improve the inner-loop optimization. One of the hypotheses of this work was that this trajectory contained useful information for solving the surrogate objective. To explore this hypothesis we apply a momentum-based step to the parameters before starting the inner-loop optimization, aiming to provide a better initialization for the inner loop optimization. Here, by ‘better initialization’ we mean an initialization closer to a solution of the surrogate objective, hence easier to optimize for. The reviewer's concern about off-policy distribution shift is valid, although, using the biased initialization does not affect the data collection nor the establishment of the surrogate objective / trust region. The trust region size (defined by $\epsilon$) is the principal component of PPO that moderates off-policy distribution shift. As biased initialization does not affect the trust region size, it should not directly increase the off-policy distribution shift.

3. Brax and Jumanji were chosen because they are implemented in JAX, which allows for high-performance parallel computation and end-to-end compilation of RL training. This enabled us to perform extensive experimentation and hyperparameter sweeps within a reasonable computational budget. Furthermore, these simulators offer sufficiently diverse tasks, making them well-suited for evaluating the generality of our approach. Whilst there are more extensive suites, we felt the ability to tune a strong baseline against which we could reliably measure progress took precedence.

4. Yes, each trial represents a distinct choice of hyperparameters, and each agent represents a random seed. The performance of each trial is averaged over four agents (i.e., four random seeds) trained using the same hyperparameters. Learning curves for the baseline w.r.t environment timesteps are provided in Figure 9, with individual task learning curves in Figures 22 - 24; the final performance of the baseline can be determined from these plots. Figure 10 shows the performance during baseline tuning using the Tree Parzen estimator; the x-axis represents the trial number and the y-axis represents the mean return of the 4 agents trained for that trial. In this figure we observe the best performing trial (highest point on y-axis) increases as more trials are completed (moving right on the x-axis), albeit with diminishing improvements for many tasks. We have amended the caption to make this clearer for the reader, and added a red line showing the highest performing trial thus far.

5. The optimality gap is a metric provided by the RLiable library (https://github.com/google-research/rliable). Using the notation of their work, optimality gap is defined as $\gamma − mean$, where $\gamma$ is a defined value of ‘optimality’. Given that we normalize our return by the minimum and maximum achieved across all agents during the sweeps, and set $\gamma = 1$, the optimality gap plot simply mirrors the mean plot. Since this metric is redundant in our setting, we have removed it from the manuscript.