Efficient Off-Policy Learning for High-Dimensional Action Spaces

Dear Reviewer urYk,

We appreciate your thoughtful feedback and your evaluation of our work. We are glad that you recognize the value of providing a modern take on weighted importance sampling in the context of deep RL, and that our work might help expand the design space for RL algorithms. One of our goals was indeed to highlight how combining classical concepts with recent advancements in RL can yield effective algorithms, and we appreciate that this has been acknowledged.

Novelty and Comparison to State-of-the-Art Algorithms

While the foundational idea of weighted IS is indeed well-studied in tabular and linear settings, our primary contribution lies in its adaptation to the deep RL setting with V-functions and the demonstration of its competitiveness against established deep RL algorithms, which has not been explored in the literature. Furthermore, we provide theoretical insights regarding the variance behavior of different methods and the connection to linear and tabular methods.

We do not seek to replace existing methods altogether, but rather, as the reviewer also mentions, to offer a new perspective and practical insights into deep off-policy RL that may inspire further innovation. Instead of relying on the usual Q-functions, it focuses exclusively on V-functions. A practical consequence of this is better suitability for high-dimensional action spaces, as this eliminates the need to learn over the joint state-action space, making the method less susceptible to the curse of dimensionality and more effective at updating the critic independent of the action. Our initial focus has been to demonstrate the general feasibility and benefits of such a setting on original baselines such as SAC, which typically serve as the basis for various improvements and variations. This involves providing theoretical insights into variance and methodology, while drawing parallels to the linear domain, and finally translating these ideas into a practical and functional algorithm. We therefore see methods such as TQC, TD7, BRO, iQRL, etc. as orthogonal work. In this sense, as also mentioned in our future work, we are interested which ideas from Q-function-based methods, such as distributional critics, critics ensembles, prioritized experience replay, network resets, quantization, or different critic/policy architecture choices, like BroNet, also translate to Q-function-free approaches. We tried to incorporate this motivation more clearly into our revised introduction.

Generality and Overfitting Concerns

In our revision, we added additional experiments on the $39$-dimensional MyoHand from MyoSuite using the same setting as for DMC's dog to cover a larger variety of tasks and provide evidence outside of the standard locomotion tasks. Similar to the initial submission, where we could show competitive performances on DMC's dog, Ant and Humanoid, our method is also able to learn consistently for MyoSuite and achieves a significantly better aggregated performance. Additionally, to mitigate potential over-tuning to specific environments, we followed a consistent training protocol across all environments and limited environment-specific hyperparameter tuning.

I am surprised by the effectiveness of IS in a setting with high dimensional actions, where the ratios may have particularly high variance. Clipping can help avoid overly large values, but what about overly small values? Do the ratios remain close to one especially as the replay buffer grows and the behavior and current policies become more distinct?

We found that during learning the importance weights for older samples indeed decrease over time as the buffer grows. However, the impact of smaller importance weights is limited due to the structure of the loss that is induced by weighted importance sampling. As shown in Figure 1, the WIS loss behaves differently to e.g. the V-trace in that the loss is scaled, which means that the gradient for these samples is also scaled accordingly, limiting the potential impact they can have on learning stability.

Dear Reviewer mtMd,

Thank you for your detailed feedback and insightful comments. We are pleased that you found value in the simplicity, theoretical justification, and comprehensive experimental evaluation of our method. Your recognition of the ablation studies, confidence intervals, and the intuition provided by Figure 1 is particularly encouraging, as we aimed to present a clear and rigorous evaluation.

Motivation

We acknowledge that our paper could better consolidate its motivations. We do not seek to entirely replace existing methods, but rather to offer a new perspective and practical insights for deep off-policy RL. Rather than relying on the usual Q-functions, it focuses on V-functions exclusively. A practical consequence that follows is better suitability for high-dimensional action spaces since this eliminates the need to learn over the joint state-action space, making the method less susceptible to the curse of dimensionality, and more effective in updating the critic regardless of the action. Our initial focus was on demonstrating the general feasibility and benefits of such a setting against original baselines, like SAC, which typically serve as the basis for various improvements and variations. This includes providing theoretical insights regarding variance and methodology while drawing parallels to the linear domain, and lastly transferring these ideas into a practical and functional deep learning algorithm. As for V-trace, while our main goal is not necessarily to simply improve it, V-trace is the best directly comparable baseline for our setting. However, as discussed in the main paper, it naturally has certain shortcomings that can be avoided due to the different foundation of Vlearn. Finally, while we believe this shift away from state-action value learning to state value learning could potentially have implications for scaling RL, we do not consider this the main goal of this work. We tried to incorporate this motivation more clearly into our revised introduction.

Baseline Comparison

We see this work as a "vanilla" approach of an alternative direction to the main branch of current deep off-policy RL research. We think making changes to Vlearn, such as replacing target networks with BatchNorm/LayerNorm for stabilization, distributional critics, or other choices, are orthogonal to our line of work and are interested in our future work which ideas for Q-function-based methods also translate to Q-function-free approaches. For these reasons, we believe original baselines, such as SAC, provide a fair comparison. We have now also added additional evaluations on the $39$-dimensional MyoHand from MyoSuite using the same setting as for DMC's dog to cover a larger variety of tasks and provide evidence outside of the standard locomotion tasks. Similar to the initial submission, where we could show competitive performances on DMC's dog, Ant and Humanoid, our method is also able to learn consistently for MyoSuite and achieves a significantly better aggregated performance.

Citations and Clarifications

In the updated version, we revised the abstract and added citations and minor improvements as mentioned by the reviewer. We also did clarify, which "target" we are referring to. However, we wanted to point out that l.177 seems to be a misunderstanding, we do not refer to the current estimate as target. The intended meaning of this sentence was "[...] interpolating between the Bellman target and the current value function $(1 − \rho_t)V_\theta (s_t)$ or target value function (1 − \rho_t)V_\bar\theta (s_t) in the 1-step case". This was an effort to generalize for cases with and without target network, but we agree this might be confusing and removed it.

Importance Sampling and Replay Buffer

Although our setting is off-policy, the importance sampling in general remains relatively similar to that of the on-policy setting. The changing target distribution is also present when updating the policy via importance sampling, e.g. in PPO. At the same time, the mini-batching, which is typically used in on-policy methods, is effectively acting as a small temporary replay buffer. As a result, we can expect similar instabilities as in those methods since the policy is updating rather slowly and is naturally bounded by leveraging a trust region via TRPL. However, one additional source of instability, which is significantly more relevant for the off-policy setting, is the "age"/staleness of samples, i.e. the discrepancy between the two policies used for importance sampling. As shown in our ablation results, leveraging the weighted importance sampling loss, importance weight clipping, etc. can alleviate these issues to ensure stable training.

Thank you for answering my questions. I will happily confirm my initial review.

Dear Reviewer ej3J,

We appreciate the thoughtful feedback and the recognition of our work’s strengths. We are glad to hear that the clarity of our writing and the rigor of our experimental evaluation were well-received. Your comments on Figure 1 and the transparency regarding limitations are particularly encouraging.

PPO-style Loss vs TRPL

The trust region used by PPO effectively depends on the various implementation details present in PPO. This makes it more likely for the policy updates to be too large, i.e. violate the KL trust region, potentially leading to unstable learning (cf. the analysis in [1,2]). Paired with the off-policy setting and the increased variance due to the importance sampling, the overall gradient direction becomes less reliable, making these larger updates more risky. In Appendix C, we have also evaluated the PPO loss for the lower dimensional environments, for which the PPO loss works reasonably well. However, due to its heuristic nature, we found that it does not scale well to more complex problems.

Performance in Noisy or Non-Stationary Settings

While our current experiments focus on stationary settings, we anticipate that Vlearn may exhibit sensitivity to noise similar to other existing approaches, since this often affects all parts of training, including representation learning, credit assignment, etc. While trust regions such as those offered by TRPL can potentially alleviate some of these problems, they are unlikely to be sufficient.

References

[1] Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann. Differentiable trust region layers for deep reinforcement learning. In International Conference on Learning Representations, 2021.
[2] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. In International Conference on Learning Representations, 2020. URL http://arxiv.org/abs/2005.12729.

Dear Reviewer HaA8,

We appreciate your positive feedback on our paper and are grateful for your insights. We are pleased to hear that you found our experimental results promising and that the theoretical analysis provided valuable context for our design choices.

High-Dimensional vs Low-Dimensional Action Spaces

High-dimensional action spaces often require more nuanced decision-making and pose a more complex learning problem in the case of Q-function-based approaches, such as SAC. They require learning over the joint state-action space, making them more susceptible to the curse of dimensionality. In contrast, low-dimensional action spaces typically only add a small amount of complexity when comparing V-function and Q-function learning, making it easier for algorithms like SAC to optimize effectively. Additionally, when only using a V-function in the off-policy setting importance sampling becomes necessary, which naturally adds additional variance to the training. From a simplified view, one could argue that this effectively boils down to a trade-off between the complexity of learning the critic vs the amount of additional variance.

We want to thank all reviewers again for their time.

We see our approach as a "vanilla" V-function method, and we wanted to illustrate its advantages over existing "vanilla" Q-function-based methods (SAC) and existing off-policy V-function methods (V-trace) - we have now added the results for the latter to MyoSuite as well, and Vlearn again clearly outperforms V-trace.