Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference

Thank you for your review our paper. We really appreciate your praise of our novel theoretical contributions and experiments. We have attempted to address each of the comments you made in the weaknesses section of your review below:

Testing Strategies Other Than DQN: As our main contribution is focused on policy inference, we only implement DQN policy learning for our method and all baselines because it is a very generic choice for the learning method. We were wondering if you could elaborate a bit more on this point. Do you have any specific algorithms in mind that you would like to see us try? We are trying to get a sense of what questions you feel are currently unanswered regarding the choice of only considering the DQN for learning so that we can properly address this concern.

Parallel Updates Baseline: Thank you for this helpful suggestion, it is indeed an interesting ablation to consider within the discourse of our paper. We have added the result of new experiments to address this comment in Figure 4a of the revised paper. For this experiment with the 100M parameter model, we had leveraged 33 learning processes with a batch size per process of 16 in order to keep up with the environment. As such, for the parallel update baseline, we accumulate gradients from each process and only update the policy parameters with the result of this big batch once all processes have been computed. This implies that the effective batch size is 33 times larger and that there are 33 times fewer parameter updates over the course of learning. The gradients from the larger batch size does not seem to be effective enough at integrating the data to make up for so many fewer updates to the model. This is in line with our discussion of the decision to consider asynchronous learning in Section 3.1. We have experiments for this ablation also running right now that we will add to Figure 4b. Unfortunately, we were not able to get access to the compute resources needed to get them done in time for the paper revision deadline, but we will definitely include this for the camera ready draft.

Application to Multi-agent Systems: Staggered asynchronous inference can definitely be applied to multi-agent systems and we believe that this is a very interesting direction for future work. Based on your suggestion, we have added Appendix C in which we discuss the key challenges that should be considered to enable real-time multi-agent use cases in the future starting at line 1173.

Thank you for taking the time to provide a detailed review of our paper. We really appreciate the way you highlighted our novel contributions and potential impact on the field. We have attempted to address each of the questions and concerns that you raised in your review below.

Only Depends on the MDP (Start of Section 2): Thank you for pointing out how confusing this phrasing was in our submission. We were definitely not trying to say this is the case for the asynchronous interaction framework. Rather, what we were attempting to convey is that this is an unrealistic implicit assumption in papers that consider the standard RL interaction framework. This is the exact opposite of the case for the asynchronous interaction setting we consider in this work. Indeed, the entire point of introducing the concept of the induced delayed semi-MDP is because the time between decisions is directly controlled by the agent and we would like to thus understand the tradeoffs associated with particular choices. We have updated our discussion of this in the first paragraph of Section 2 (starting at Iine 81) accordingly.

Selecting the Default Behavior Policy: Thank you for bringing up these interesting questions regarding the design of the default behavior policy. In our work, we assume this policy is provided as part of the environment and in fact our theory considers the worst case scenario in which following it always leads to suboptimality. As such, our primary focus in this paper is on avoiding use of it altogether rather than trying to design it to be a safer option. However, we believe that exploring these questions in more detail would be a very interesting avenue for follow up work inspired by the formulation of our paper. We have added a discussion of challenges and avenues for future research related to designing safe default behavior policies in Appendix C.

Impact of Swapping Noop with the Worst or Best Action: Our worst case bound for inaction regret in Equation 3 of Theorem 1 relies on the idea that the default policy always follows suboptimal or worst case behavior. If the default policy always followed the best action, the optimal policy would be to never act in the environment, which is a scenario of limited interest to realtime RL scenarios. It is important to point out that changing the default behavior would have no impact on the results of our method using asynchronous inference, as we supply enough inference processes to act at every step and thus never follow the default behavior policy. For the sequential interaction baseline, whether other action choices would help depends on the nature of the game. For example, random actions would help with progress through episodes but lead to less success within those episodes than “noop” actions as i.e. in Pokemon sometimes actions cannot be reverted and guarantee failure after they are taken. It is a nice property of “noop” actions in Pokemon that they never have irrevertible consequences, which is a major difference with the Tetris scenario.

Questions 1 and 2 are not hyperlinked while Questions 3 and 4 are: We really would like to address this when we update the paper, but feel like there must be a miscommunication. We re-downloaded our submitted draft on different computers and opened it with different applications. There is no hyperlink on any of the questions and the formatting is consistent in latex. Are you suggesting that we should add hyperlinks? Are you using a particular software to open it? Is it consistent for you in the updated draft or is there still an issue?

Comparing Statements of Question 1: Thank you for pointing out that this was confusing. We conflated these concepts because from our perspective achieving higher action throughput and extended inference times are two sides of the same coin as the action throughput is equal to the inference time divided by the number of staggered asynchronous inference threads for our approach. So the point we were trying to make is that better performance in these environments is not simply achievable with higher throughput and also requires the agent to learn effectively to move on. We have rephrased the two versions of the question now so as to avoid confusion about consistency of the meaning.

Thank you for your insightful review of our paper. We really appreciate you highlighting the strength of our writing, the applicability of our method for real-world systems, and the quality of our empirical results. We wanted to make sure that we addressed each of the questions and concerns you raised in our response.

Artificially Injecting Noise: Thank you for suggesting this analysis. This is a very interesting idea. Unfortunately, we were not clear on how we could do this in a reasonable way for complex environments from our submission such as Pokemon and Tetris. If you have any suggestions on how this could be implemented, we would be happy to give it a try. We did, however, add some additional experiments using a realtime simulation of three Atari games in Figure 6 of our revised draft. One trend that we thought somewhat spoke to your idea is the different scaling behavior we see for asynchronous interaction on the games Boxing and Krull. In both cases, we see much better performance than sequential interaction at large model sizes, but there is a big difference in the degree to which asynchronous methods can be scaled up while still achieving human-level performance. In Boxing, we see it is not possible to achieve human-level performance with even a small amount of delay, performing worse than human-level for anything with 3 or more steps of delay. Meanwhile, for Krull we find that it is possible to maintain human-level performance with as many as 750 steps of delay. This makes sense in the context of these games as the opponent has a very stochastic policy in Boxing whereas the dynamics of Krull are far more predictable.

Do Algorithms Achieve Sublinear Regret?: Thank you for bringing up this constructive comment as well. It is a natural question arising from the result we demonstrate in Remark 2 that fits really well with our overall discourse. We have added a footnote about this surrounding Remark 2 in the main text and have added Section B.6 of the Appendix to discuss this point in detail. The main conclusion is that combining Algorithms 1 and 2 with Q-Learning that leverages appropriate optimism and learning rates can guarantee a regret upper bound that is sublinear in time for deterministic environments. More generally, regret will be sublinear in time as long as delay regret is sublinear in time.

Connection to Action Chunking: This was another very insightful comment. As we have now clarified in Section 2, we are focussed on policies that produce one action at a time in this paper. However, the action chunking approach is indeed quite relevant as a point of comparison. We have added a footnote about it to the main text and have added Section B.5 of the Appendix to discuss the action chunking approach in detail. One takeaway we highlight is that while there may be some circumstances in which action chunking can be used to eliminate inaction, this is not the case in general (as we can demonstrate for staggered asynchronous inference in Remark 2). Our other key conclusion is that action chunking leads to an action space that grows exponentially with the growing chunk size and that this in-turn leads to an exponential increase in the lower bound on regret from learning.

The Case When $\pi(s_{t-1}) \neq \pi(s_{t})$: We were not totally sure we understood the question, but one point to clarify is that our policies learn to accommodate for the lag in action execution. For example, if action inference takes $k$ steps, our asynchronous inference policies learn to choose $a_{t+k}$ appropriately given $s_{t}$ rather than $a_{t}$. As a result, it is possible to learn a very different $a_{t+k}$ than the $a_{t}$ that would be learned in the case of no delay. Please let us know whether this explanation addresses your confusion.

Larger $N_{\mathcal{L}}^∗$ Values with Smaller Batch Sizes: To clarify, $N_{\mathcal{L}}^∗$ represents the amount of processes needed to learn from all incoming environment transitions. As a result, while smaller batch sizes certainly do take less time to compute, they do not quite take linearly less time to compute (due to decreased opportunity for parallelism) to make up for the fact that they are processing fewer items. To illustrate when we mean, for the 10M parameter model the update time for batch size of 8 is 1.898 seconds, the update time for a batch size of 16 is 3.234 seconds, and the update time for a batch size of 32 is 6.061 seconds. But if we consider the update time per transition, a batch size of 8 corresponds to 0.237 seconds, a batch size of 16 corresponds to 0.202 seconds, and a batch size of 32 corresponds to 0.189 seconds. As a result, more processes will be needed to keep up with the throughput of transitions from the environment with smaller batch sizes to accommodate for the greater amount of time needed to learn from a transition in each process. Please let us know if this still doesn’t make sense and we would be happy to clarify.

Thank you for your review of our paper. We really appreciate the way your review highlighted our theoretical contributions and the presentation of our paper. You also made some great suggestions in the weaknesses section that we wanted to make sure to address in our response and revisions.

Evaluating More Experimental Settings: This is a great suggestion. We have added a new set of experiments to augment our experiments provided in the submitted draft in Figure 6 of our revision. We considered a realtime simulation of three Atari games (Boxing, Krull, and Name This Game) following a similar procedure to our simulation of the Game Boy. The particular games were selected to allow for learning an agent that performs on-par with humans within 2,000 episodes of training without the presence of delay. This way we can clearly see the degradation of performance as the degree of delay and inaction are increased with large models. Our experiments in these new domains further corroborate our key findings in Tetris and Pokemon, demonstrating the generality of our proposed approach.

Why sequential learning fails as network size increases: Sequential interaction fails as the network size increases because more inaction and delay is experienced when the action inference time becomes larger. For example, in a ResNet model with sequential interaction, a non-noop action is only taken every 5 steps for the 1M model, every 8 steps for the 10M model, every 70 steps for the 100M model and every 596 steps for the 1B model. In all of our experiments, we plot the performance of a policy that takes all noop actions and we see that larger models converge towards this behavior within the sequential interaction framework. Please let us know if this clarifies your concern. We were not really sure what kind of evidence you were additionally looking for in terms of qualitative examples of behavior.

Variations in the number of layers, in addition to the number of channels: This was another great suggestion. We added an ablation to our new experiments in Figure 6 with a deeper variant of the ResNet architecture we consider leveraging 30 convolutional layers (as opposed to 15 convolutional layers). As a result, the channel number multiplier needed to get to the same number of parameters is lower and the inference time at the same number of parameters is higher. As such, there is even more delay and inaction for algorithms to deal with at a particular model size. Predictably, we see that the main effect is that the performance of the sequential interaction baseline falls off even faster.

Experiments with high inference time variance to validate Algorithm 2: We actually did try to address this point in Figure 7 of the submitted draft, which we have updated for increased clarity. The idea is that although the neural network forward propagation time has low variance, this time is quite different from the potential speed of a simple random number generator during exploration steps. As such, we compare the throughput at $\epsilon=0$, where all actions are taken by the neural network and the variance in action inference times is low, with the throughput at $\epsilon=0.5$, when half of all actions are very fast random actions. In line with our theoretical predictions, we find that Algorithms 1 and 2 require the same number of total processes when the inference time variance is low and that Algorithm 2 is more efficient when the variance is high. We explain this starting at line 454 in the revised draft.

Please let us know if you have any additional questions or concerns based on our explanation and we would be happy to clarify during the discussion period.