Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

We are grateful to the reviewer for their time and commitment to evaluating our paper. We're glad the reviewer noted the importance of the problem setting, the diversity of tasks, and the use of 30 seeds per task. They also appreciated the clarity and writing quality. We're also pleased to read that the reviewer thinks our analysis of normalization effects on actor-critic methods is important and timely. Here, we first summarize the key points raised by the reviewer and respond to them.

Clarification of Contributions

Is AVG essentially SAC without Clipped Double Q-learning but with added feature normalization, state normalization, and return scaling?

It is similar, but not the same. Here, we outline the components of both algorithms:

In addition, SAC is off-policy, whereas AVG is on-policy. SAC samples an action and stores it in the buffer. Unlike AVG, SAC's action is not reused to update the actor.

We appreciate the reviewer's constructive criticism about the clarity of the contributions. We will incorporate their feedback in our final manuscript.

Presentation of AVG

Design choices such as orthogonal initialization, squashed normal policy, and maximum entropy formulation, which is standard in modern SAC implementations, can be condensed into a single paragraph

We intended to outline every component utilized in our algorithm to enhance reproducibility. We agree with our reviewer that this information can be presented briefly.

Comparison with Other Normalization Techniques

Can effects of state normalization and return scaling be achieved by using layer normalization on the actor/critic modules?

As requested, we conducted additional ablations studying the effect of alternatives such as layer norm and RMS norm (Fig. 3 of PDF). In this comparison, our proposed techniques perform best when used with AVG.

Comparison of strategies for state and return scaling

We chose Welford's online algorithm for normalizing observations since it is unbiased and maintains statistics over the entire dataset. In limited experiments, weighted methods biased toward more recent samples failed to work well in our experiments.

Schaul et al. (2021) showcase the perils of clipping or normalizing rewards. Hence, we chose their simple approach, which scales the temporal difference error using a multiplicative scale factor.

We agree with the reviewer that a different set of normalization techniques can potentially obtain similar, if not better, results. We only aim to highlight this issue and propose easy-to-use solutions.

Argument for using orthogonal weight initialization

Our use of orthogonal weight initialization is consistent with popular implementations of SAC and TD3. However, empirical evidence is left for future work.

Typo in Algorithm 1

We thank the reviewer for catching this typo in our critic update rule in Algorithm 1. We will fix this in our final manuscript.

Answers

Does AVG use two critics?

AVG uses only one critic, which is an unclipped Q-network.

Is there a reason to consider a single task from the DMC benchmark?

We consider DMC to determine the effectiveness of AVG in sparse reward environments. Additional results from environments such as Ball in Cup and Finger Spin will be included in the final manuscript.

How does the fact of being incremental affect the RP policy gradient? Does this on-policiness negatively affect the convergence properties?

SAC's convergence argument results from Soft Policy Iteration (SPI) in the tabular setting. In this case, the critic converges to the true soft value function of the actor before the actor is updated. Under this regime, SPI converges to the optimal soft policy (Haarnoja et al., 2018). If we take the two-timescale perspective (Konda & Borkar, 1999; Konda & Tsitsiklis, 1999) on SAC, it can be seen as approximate (soft) policy iteration, which can serve as theoretical motivation for SAC. This argument would work for AVG as well.

It is difficult to directly compare the theoretical properties of the actual implementations of AVG and SAC, given that they have many varying components. However, we may gain insights from analyzing algorithms with similar structures that only vary in the on-policiness/off-policiness.

Following this direction, Xiong et al. (2022) show that for Deterministic Policy Gradient (DPG), up to the same system error that necessarily exists in actor-critic algorithms, on-policy DPG and off-policy DPG (with off-policy correction) have the same sample complexity of $\mathcal{O}(\epsilon^{-2})$ to achieve an $\epsilon$-accurate stationary policy. Note that existing implementations of deep off-policy algorithms do not include off-policy correction (unlike in Xiong et al. (2022)).

We validated that these results can be extended to the case of reparametrized policy gradients, suggesting that changing from an off-policy distribution to an on-policy distribution for updating the critic would not induce a negative impact. In addition, reducing the batch size would impact sample complexity, but it is necessary for applications involving edge devices.

Overall, a more careful theoretical understanding of AVG is essential and is left for future work.

Does AVG use target networks?

AVG does not use target networks.

We agree with the reviewer that it is indeed interesting that target Q-networks negatively impact incremental algorithms. We provide additional results in our PDF (Fig. 2) that show that SAC-1 can still fail when normalization and scaling tricks are incorporated.

We believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.

We are grateful to the reviewer for their time and commitment to evaluating our paper. We are pleased to read that the reviewer thinks our paper is well-written and that the problem we focus on is important to the continual learning and adaptive learning communities as well as for applications in search-and-rescue scenarios and ocean/sea exploration. Here, we jointly address the reviewer's points and questions in the order of priority suggested by the reviewer.

Answers

Comparison with other incremental/continual learning methods (mentioned in the paper) in the low compute regime (The authors may have to again use the same normalization tricks).

We have results incorporating normalization and scaling with incremental actor critic (IAC) listed in Appendix B.2. For further evidence, we also provide additional results that compare AVG and IAC with normalization and scaling in our global response PDF (see Fig. 2). We demonstrate that IAC also benefits from normalization and scaling.

Making the comparison with SAC-1, SAC-100, TD3-1 with the same normalizations.

The reviewer raised concerns that the comparisons with SAC-1, SAC-100, and TD3-1 may be unfair since they do not use similar normalization and scaling techniques to stabilize learning. As requested, we have additional results incorporating normalization and scaling with SAC-1, SAC-100 and TD3-1 and comparing them against AVG. Please see our results in the global response PDF (Fig. 2). Notably, SAC-1 combined with normalization and scaling performs effectively on two tasks but fails or even diverges on the others.

We would like to emphasize that the primary purpose of these experiments (Fig. 2 in our paper) is to show that existing methods cannot be naively used off-the-shelf in the incremental learning setting. Currently, there is no clear evidence in the literature, and as such, the community remains unaware that these batch methods fail catastrophically in the incremental learning setting. Therefore, our experiment was justified, given the research question we were addressing.

An ablation with relatively low batch and buffer sizes.

This is an interesting question where the reviewer asks how best to utilize a resource-constrained device. This study needs to be carefully designed and conducted appropriately. We agree with the initial direction provided by the reviewer as well:

“An ablation study how much low compute device could be exploited would make the paper stronger. For example, a study of incremental learning where having a buffer size n, batch-size m and varying sizes while still being able to work on the low compute device would provide more insights on the incremental learning problem in such devices.”

However, given the time constraints, this is out of the scope of our paper. That said, we definitely believe this should be pursued in future work!

We believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.

We are grateful to the reviewer for their time and commitment to evaluating our paper. We are pleased that the reviewer recognizes the value of eliminating replay buffers and target Q-networks to develop simpler, more computationally efficient learning algorithms. Below, we address the points raised by the reviewer.

Sensitivity to Hyperparameters

We agree with the reviewer that deep RL algorithms are notoriously sensitive to hyperparameters, including the random seed. One way to mitigate hyperparameter tuning is to identify a single set of hyperparameters that is robust across a wide range of tasks. While this set may not be optimal for each task individually, it may perform well across many tasks. Popular algorithms like DreamerV3, TD-MPC2, SAC, and PPO often showcase their robustness by using a single hyperparameter configuration that works well across multiple benchmark environments.

Out of the 300 configurations presented in the paper, one consistently ranked in the top 10 across four environments and achieved effective performance, coming within 80% of the best performance in each. We conducted additional experiments with this configuration, running 30 different seeds for 10 million steps. The learning curves for these experiments are shown in Figure 1 of the global response PDF. The hyper-parameter configuration is provided below:

We hope this addresses the reviewer's concerns.

Sample Efficiency

We acknowledge that there is potential to improve AVG's sample efficiency. Off-policy methods like Soft Actor-Critic (SAC) indeed achieve higher sample efficiency by utilizing replay buffers, which allow for multiple uses of each sample. However, these methods are often not suitable for deployment on microcontrollers and resource-limited systems due to their memory requirements and computational complexity.

Our current focus is on fixing incremental deep policy gradient methods, which we consider a crucial first step. Improving the sample efficiency of these incremental methods remains an important area for future research.

Answers

My main concern is that deep RL algorithms have been shown to be very sensitive to hyperparameters, network size, reward scaling, and even random seeds! And now that we have AVG with incremental updates, this might exacerbate this issue.

We would like to emphasize that we did not perform any hyper-parameter tuning for the real robot experiments. Due to the significant cost and time requirements associated with real robot experiments, it is impractical to conduct hyper-parameter tuning directly on real robots. Instead, we selected an effective hyper-parameter configuration based on our simulation experiments and applied it to both UR-Reacher-2 and Create-Mover tasks (see Fig. 8 of paper). Additionally, we have previously mentioned that we identified a set of hyper-parameters that perform well across various tasks, further supporting the robustness of our chosen configurations.

Effect of network size

It is important to clarify that our current work does not demonstrate or claim any sensitivity of our algorithm to network size. All our experiments involve neural networks with two hidden layers, 256 units each. Investigating the impact of network size on our algorithm's performance and efficiency remains an open question for future work.

Is there a trade-off between not storing experiences/target models and robustness?

A large replay buffer can place a heavy memory burden, especially for onboard and edge devices with limited memory. Therefore, we need computationally efficient alternatives to replay buffers that can help consolidate learned experiences over time. So, we agree that there is a trade-off here. Lan et al. (2022) explore this trade-off by introducing memory-efficient reinforcement learning algorithms based on the deep Q-network (DQN) algorithm. Their approach can reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network while only using small replay buffers. We recognize that the question is intriguing and remains an open area for future research.

We believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.

References

• Lan, Q., Pan, Y., Luo, J., & Mahmood, A. R. (2022). Memory-efficient reinforcement learning with value-based knowledge consolidation. arXiv preprint arXiv:2205.10868.

We are grateful to the reviewer for their time and commitment to evaluating our paper. We are pleased that the reviewer thinks our proposed method, AVG, is a novel, "well-grounded algorithm advancement" that tackles an important challenge in real-world applications. We are also happy that the reviewer appreciates the extensive set of experiments both in simulation and on real-world robots. Here, we first summarize the key points/concerns raised by the reviewer and respond to them.

The Motivation for Choosing AVG over IAC

Our motivation for introducing AVG is two-fold:

• Reparameterization gradients offer an alternative approach to estimating the gradient and have been observed to demonstrate lower variance in practice (Greensmith et al. 2004, Fan et al. 2015, Lan et al. 2021).
• There is a gap in the existing literature, as no incremental RP gradient method currently exists.

We mention this in the introduction, but we can state this more clearly in the final manuscript.

The Essentiality of Normalization and Scaling

From Figure 3, it is unclear whether normalization and scaling techniques are essential for incremental deep policy gradient learning, given that IAC without these techniques shows reasonable performance.

We believe there is some misunderstanding here. IAC performs very poorly in Figure 3, exhibiting high variance and low mean performance. The vision-based reacher task used here is an adaptation of the Deepmind Reacher task with a reward of $-1$ every step until the arm's end-effector reaches the goal. The return indicates that IAC has failed to learn a good policy that can successfully reach the target. We will further clarify this point in the final manuscript.

Normalization and scaling are not necessary for simple environments like Reacher-v4 and Acrobot-v4. However, they are crucial for robust performance on complex simulation benchmarks and real-world robot tasks.

Answers

Can the authors provide comparison and detailed analysis on LR and RP? In Fig.14 at Sec B.2, normalization & scaling techniques on IAC also indicate great performance for LR-based method.

Yes. Normalization and scaling do benefit IAC as well. We have included additional experimental evaluation that compares AVG and IAC, both benefiting from normalization and scaling (Fig. 2 in global response PDF).

How about normalization & scaling techniques applying to SAC-1?

Since there are multiple implementations of SAC with different design choices, we first briefly list the important components used in our implementation of SAC:

• One actor network.
• Two Q networks (i.e., double Q-learning).
• Two target Q networks.
• Squashed normal policy.
• Automatically tuned entropy coefficient.
• Replay buffer.

With SAC-1, we only get rid of a replay buffer, using only the most recent sample of updating all the networks. SAC-1, combined with normalization and scaling, performs effectively on two tasks but fails or even diverges on the others (Fig. 2 in global response PDF). Target networks may hinder incremental learning. Removing these will bring the algorithm closer to AVG.

Take into consideration the application scenario, instead of learning from scratch, it might be beneficial to further discuss on the performance of different incremental learning methods with pre-trained policies.

In this work, we priortized learning from scratch to keep the study focused. We aim to explore this interesting direction in future work.

Can the authors add explanations of the relation between the proposed method and real-world reinforcement learning methods, e.g. [1]?

This is an interesting question. Smith et al. (2022) demonstrated real-time learning capabilities on a quadruped locomotion task. They use a batch method with a replay buffer that combines DroQ [2] and layer normalization [3] with SAC. They utilize a laptop with NVIDIA GeForce RTX 2070 GPU for onboard inference and learning. There are some notable differences compared to our setup:

• Their learning was not onboard as the laptop was too bulky to be carried on the robot. In their videos, it can be clearly seen that the robot is tethered to a laptop using a wired connection. In contrast, our mobile robot experiments involve a small edge device directly attached to the iRobot Create2.
• Our edge device, Jetson Nano, is significantly less powerful than their laptop. Their approach involves highly optimized JAX implementations and other code-level optimizations that reduce the computational time of an otherwise intensive batch gradient method. In contrast, our method, implemented in PyTorch, is incremental and computationally cheap by design, without many code-level optimizations.
• [1] also constrains the joints to limit the robot's exploration, enabling quick learning. In contrast, we do not apply this constraint and demonstrate that AVG can perform well without extensive engineering.

We acknowledge that there can be trade-offs between batch and incremental methods depending on memory availability and compute resources onboard a robot. With highly optimized software and hardware, running constrained variants of batch methods might be possible. We only suggest that our incremental methods would be much more amenable to real-time learning compared to current batch methods.

We believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.

References

1. Smith, L., Kostrikov, I., & Levine, S. (2022). A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860.
2. T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka, "Dropout q-functions for doubly efficient reinforcement learning," International Conference on Learning Representations (ICLR), 2022.
3. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.

Thank you for the effort, and I am pleased with the added explanation and experimental comparisons. I also agree with the authors that incremental methods would be much more amenable to real-time learning compared to current batch methods. My concern left is

Yes. Normalization and scaling do benefit IAC as well. We have included additional experimental evaluation that compares AVG and IAC, both benefiting from normalization and scaling (Fig. 2 in global response PDF).

Normalization and scaling are not necessary for simple environments like Reacher-v4 and Acrobot-v4. However, they are crucial for robust performance on complex simulation benchmarks and real-world robot tasks.

Since IAC+ "IAC with normalization & scaling" do benefit from normalization & scaling, would it be beneficial to conduct all experiments with IAC+ (or AVG-) as one baseline on complex simulation benchmarks and real-world robot tasks (e.g. Fig.3's)? ( To make the contribution of AVG vs. normalization & scaling techniques more clear.)

I lean towards keeping the current score of weak accept given the above major experimental consideration.