Time to Truncate Trajectory: Stochastic Retrace for Multi-step Off-policy Reinforcement Learning

While off-policy reinforcement learning methods that is based on one-step temporal difference learning have shown to be promising for solving complex decision making problems, multi-step lookahead from behavior policies is still challenging by the discrepancy between behavior policy and target policy. Several recent works have addressed this challenge by introducing the coefficients to correct the discrepancy such as Retrace and evolving the behavior policy in a similar way to conservative policy iteration such as Peng's $Q(\lambda)$. However, both methods do not universally work well by the policy evaluation error caused by the value from the later part of a long trajectory data. In this work, we propose a stochastic truncation method which modify the correction coefficeints of Retrace into the sequence of Bernoulli random variables to remove the later part of trajectory, which degrades off-policy evaluation by adding the unnecessary noise. Unlike prior method for reducing the off-policy discrepancy, our stochastic truncation enjoys two strengths form the conservative and non-conservative multi-step RL methods. We demonstrate that our algorithm, time-to-truncate-trajectory (T4) outperforms various model-free RL methods.