An Empirical Study of Deep Reinforcement Learning in Continuing Tasks

• The writing is unsatisfactory, especially I believe there is a large room to improve the organization of the content. Since this work is mainly on presenting an empirical study, the purposes and structures of experiment designs and the key findings of experimental results are of the most importance to the audience. Here are a few suggestions:
  • The introduction section is a bit too long. It is now over 2 pages, which makes it difficult to convey the main thread of thoughts and findings in a clear way. A suggestion here is to add a Related Work section.
  • In Section 2, three types of testbeds are proposed one by one with a long subsection for each. My suggestion here is to add an overall description of the experiment designs and the target questions to investigate at the beginning of Section 2, then navigate to each subsection, which I think will help the reading and understanding of the audience greatly.
  • The experimental results on Atari tasks are not well discussed in the main body of the paper. I suggest the author adjust the content and include them.
• The continuing tasks considered in this work are from popular typical RL benchmarks but are still lacking in representativity. MuJoCo tasks are basically locomotion tasks, except for pusher and reacher are goal-reaching tasks, where the optimal control has a periodical feature intuitively. This cripples the significance of continuing learning to some degree. As mentioned by the authors, real-world problems like inventory management, content recommendation, and portfolio management, are story-like and indeed Atari tasks should be better options for building continuing tasks. Fortunately, the authors include 6 Atari tasks in this work but more emphasis is put on MuJoCo tasks. In addition, Minecraft could be a good choice for continuing tasks. It will be great if they are included in the future.
• For Section 2.3, it would be great to see an analysis of the learned reset control by the agents.
• In Figure 2, two concrete failure cases of learning in continuing tasks without resets are presented. However, I feel a bit lacking in further discussion on the two reasons.

Minors

• The margin below Table 6 is too narrow.

The whole premise of the work lies on the connection to real world tasks, primarily robotics. When simulation is not possible (for example, in very complex scenarios), then it is indeed of interest to deploy the robot in the real world and have it continually learn.

However, I feel that there is some mixture between theory and practice, specifically around desired metrics to measure. When deploying a robot, I don't see the reward rate being a metric of interest. There's a task, we can measure how well it is on the task. Even if it is trained in a continual setting and not a classic episodic one.

• The core methodological proposals (Section 3) represent what is, in my opinion, an incremental amendment to Naik et al.'s reward centering work. The authors show that Naik et al.'s proposals can be repurposed for the deep RL setting, but this is a logical conclusion one could draw from Naik et al.'s paper alone, and does not represent the kind of methodological novelty usually expected from a paper at ICLR.
• The paper's layout feels unintuitive. Though the authors discuss some related work in the introduction, the paper lacks a dedicated related work section in which they clearly situate their contributions in the context of others'. Equally, the authors do not formally introduce their problem and its associated notation in the main body. This becomes particularly problematic in Section 3 when they start building upon Naik et al.'s notation without prior discussion (I appreciate the authors introduce MDPs etc. in Appendix A.1, but this should come prior to the Appendix). I recommend that the authors stick to standard convention and follow something akin to: Introduction -> Related Work -> Preliminaries / Background -> Methods -> Results -> Discussion/Limitations -> Conclusions.
• The authors do not justify several decisions in the design of their problem setting. For example, in lines 146-156 they state the target position in Reacher is resampled every 50 steps, but for Pusher it is resampled every 100 steps. And in lines 294-295 they state that "[e]ach reset incurs a penalty of −10 to the reward, punishing the agent for falling or flipping". I suspect agent performance is likely sensitive to these decisions, but they are made with little justification or reference to prior work.
• The authors often use imprecise language to describe their results e.g. "decent policies" (lines 200, 202), "policy that performed reasonably well" (line 205). I would recommend that they stick to quantitative comparisons.
• In Lines 257-259, the authors state that "the agent did not perform sufficient exploration to escape from suboptimal states." Methods like SAC and TD3 are generally evaluated with the exploration noise turned off (i.e. the mean of the action distribution is used, rather than a sample from the distribution). If the authors followed this convention then this could potentially explain the results, but because they do not discuss their implementations in detail, it is unclear whether that convention was followed.

Some minor feedback:

• In Figure 1 it would be helpful if the y-axis was fixed for each environment so the reader can more easily see how performance varies across reset settings for each environment.
• Gymnasium (Towers et al., 2024) builds upon Gym (Brockman et al., 2016); I'd recommend citing the latter too.
• Spelling of "decent" in line 200
• Missing "the" in line 204