(Continue from part 1...)

Reference

[1] Otto, Fabian, et al. "Differentiable trust region layers for deep reinforcement learning." ICLR 2021.

[2] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

[3] Onur Celik, et al. "Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts.", ICML 2024. **

[4] Otto, Fabian, et al. "Deep black-box reinforcement learning with movement primitives." Conference on Robot Learning. PMLR, 2022.

[5] Li Ge, et al. “Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning.” ICLR 2024.

[6] Ba, Jimmy Lei. "Layer normalization." NIPS 2016 Deep Learning Symposium

[7] Vaswani, A. "Attention is all you need." NIPS 2017.

[8] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." ICML 2018.

================================================

If you have any further questions, please feel free to ask us.

Thank you.

TOP-ERL Authors

Dear Reviewer L7MP,

Thank you for your insightful and postive review of our paper. We have addressed your concerns as follows and included some detailed discussion in the supplementary material.

Update Log [Nov. 18]

• Address the concerns of all reviewers.

• Upload additional experiment and ablation study to Supplementary Material.

  Link: https://openreview.net/attachment?id=N4NhVN30ph&name=supplementary_material

• Revise the manuscript, with modifications marked in blue.

  Link: https://openreview.net/pdf?id=N4NhVN30ph

=======================================================

More details regarding TRPL? and LayerNorm?

Some used techniques, such as trust region constraints, layer normalization, greatly affect the performance of the proposed algorithm (see Fig. 5). The ablations of TOP-ERL, which do not use these techniques, actually achieved worse performance than previous methods. Someone may conclude that the performance improvement of TOP-ERL over baselines is achieved by using all these techniques, rather than the usage of the transformer architecture or others.

Thank you for pointing out this issue. We briefly introduce these two techniques in this reply and have added a comprehensive discussion of TRPL in Section R3 of the supplementary material, which we plan to include as an additional appendix section of our paper in the future.

TRPL

The TRPL [1] method enforces divergence constraints, such as KL-divergence, between the old and new Gaussian policies after each policy update iteration. Unlike the likelihood clipping method commonly used in PPO [2], TRPL constrains both the scale and rotation of the conditional Gaussian distribution $\pi(a|s)$ at each state $s$ in the minibatch, ensuring stable learning. If an updated policy goes beyond the trust region, TRPL projects its Gaussian parameters back to the trust region boundary and adds a trust region regression loss to the RL learning objective. This capability is essential for policies that use high-dimensional Gaussian distributions with a full covariance matrix. Previous episodic RL methods, such as [3-5], utilize TRPL for stable training, with a comprehensive ablation comparing TRPL and likelihood clipping available in [4]. We provide more details in the supplementary in Section R3.

LayerNorm

Layernorm[6] and dropout are two standard techniques utilized in the original transformer paper [7], where they were shown to be effective for sequence modeling, such as language models. In TOP-ERL, we examined these techniques and found that retaining layernorm in the transformer critic is beneficial, as it provides the only normalization in our model. However, applying dropout harms performance. We attribute this to the nature of RL data, which is generated through agent-environment interactions and inherently contains randomness. Unlike in supervised learning, RL models are less prone to overfitting; thus, adding additional randomness through dropout makes value prediction more challenging.

Design addtional ablation studies to verify the contribution of the Transformer critic?

Designing additional experiments to verify the individual contribution of the transformer-based critic network, such as TOP-ERL without the transformer, is suggested. However, I have no idea about the specific details about the exp.

We thank the reviewer for this valuable suggestion to help analyze the effectiveness of our model. In response, we conducted two additional ablation studies (introduced below) to address this concern, with their performance results detailed in Section R6 of the supplementary material. However, both ablated methods demonstrated significantly poorer performance compared to the default setting of TOP-ERL.

SAC + MP

The approach involves directly using SAC [8] to predict the MP’s parameters while retaining the trajectory generation and environment rollout procedure described in Section 4.1. We sum up the rewards collected during trajectory execution and use this as the return for the entire trajectory. Here, the trajectory is treated as a single action, and the learning objective does not incorporate any temporal information within the trajectory.

Degenerate Critic from Taking Sequence of Actions to a Single Action

The approach retains the main architecture of TOP-ERL but reduces the segment length to one in critic updates. In this case, the transformer critic degenerates to a standard critic network, as the input consists of only one state and one action rather than a sequence of actions.

(To be continued...)

Dear Reviewer L7MP,

As the rebuttal period nears its end, we would appreciate knowing if our responses have sufficiently addressed your concerns. Your feedback will assist us in making a more informed decision about our submission. Thank you for your time and consideration.

Best,

TOP-ERL Authors

(Continue from part 1...)

In online RL, however, the use of action chunks remains largely unexplored. We attribute this to challenges such as complex, unstable update rules and the difficulty in maintaining consistency for generating long-horizon, structured trajectories during exploration. Previous works, such as [9], augment SAC by incorporating an RNN as the action generator. However, most tasks in this work only predict 3-step actions. In contrast, in our work, we predict 100-500 actions using MPs.

Positional encoding for similar states at two different time steps?

I am also wondering if the positional encoding of two different states s^k_0 and s'^k_0 would always be the same (even if one is more in the future that the other one)?

Since we are working on episodic tasks with a finite horizon, the task state includes a unique timestamp, allowing the agent to differentiate the value of two similar states at different time steps. This design choice aligns with findings in the literature, such as [10]. Consequently, the positional encoding is solely responsible for distinguishing the order of the segment’s initial state and the subsequent actions within the input sequence.

Remove the Target net?

Is a target network necessary at all?

We thank the reviewer for this interesting idea. In response, we conducted an additional ablation study by removing the target network from our model and observed a performance drop, as detailed in Section R.5 of the supplementary material. Target networks are essential for stabilizing training in classical off-policy methods. However, recent work [11] has introduced novel normalization techniques that reduce the need for target networks, which could potentially be integrated into our transformer-based critic. We leave this as an open topic for future research.

References

[1] Sutton, Richard S. "Reinforcement learning: An introduction." A Bradford Book (2018).

[2] Ni, Tianwei, et al. "When do transformers shine in RL? decoupling memory from credit assignment." NeurIPS 2023.

[3] Peters, J., Vijayakumar, S. and Schaal, S., 2005. Natural actor-critic. In Machine Learning: ECML 2005.

[4] Kober J, Peters J. Policy search for motor primitives in robotics. NIPS 2008.

[5] Otto F, Celik O, Zhou H, Ziesche H, Ngo VA, Neumann G. Deep black-box reinforcement learning with movement primitives. In Conference on Robot Learning 2022.

[6] Li Ge, et al. “Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning.” ICLR 2024.

[7] Kicki P, Tateo D, Liu P, Guenster J, Peters J, Walas K. Bridging the gap between learning-to-plan, motion primitives and safe reinforcement learning. In Conference on Robot Learning 2024.

[8] Onur Celik, et al. "Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts.", ICML 2024. **

[9] Zhang, Haichao, Wei Xu, and Haonan Yu. "Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning." ICLR 2022

[10] Jackson, Matthew Thomas, et al. "Discovering Temporally-Aware Reinforcement Learning Algorithms." ICLR 2024.

[11] Bhatt, Aditya, et al. "CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity." ICLR 2024.

================================================

If you have any further questions, please feel free to ask us.

Thank you.

TOP-ERL Authors

Dear Reviewer iE4K

We appreciate your positive review of our paper and are glad that our paper’s novelty and contributions are well-understood. We address your concerns as follows to the best of our knowledge.

Update Log [Nov. 18]

• Address the concerns of all reviewers.

• Upload additional experiment and ablation study to Supplementary Material.

  Link: https://openreview.net/attachment?id=N4NhVN30ph&name=supplementary_material

• Revise the manuscript, with modifications marked in blue.

  Link: https://openreview.net/pdf?id=N4NhVN30ph

====================================================

Off-policy methods are hard to train?

The authors claim that off-policy algorithms are often more sample efficient than on-policy counterparts. However, PPO is still often the way to go when you face a new RL problems because of better stability and the need for less hyperparameter tuning. I am not fully convinced this approach won't have the same issue.

We agree with the reviewer's insightful observation. Training off-policy methods is indeed generally more challenging than on-policy methods, as off-policy approaches aim to maximize sample efficiency by reusing past experiences. This advantage, however, comes at the cost of more complex update rules, intricate model architectures, and increased hyperparameter tuning. Nonetheless, off-policy methods are advantageous in environments with costly interactions, significantly reducing the need for new samples. Our method enhances this advantage by generating structured, correlated trajectories through movement primitives and achieving efficient value estimation via a transformer critic and N-step return.

More explanation of importance sampling?

I believe the explanations of why importance sampling is not necessary could be improved.

Thank you for pointing this out. The use of importance sampling in classical off-policy RL methods arises from the need to estimate the value $Q^{\pi_{new}}(s_t, a_t)$ of a single action $a_t$ under the current policy $\pi_{new}$ using the rewards generated by future actions $a_{t+1}, …a_{t+N}$ sampled from an old behaviour policy $\pi_{b}$. The Q-function is not directly aware of these future actions, as they are not part of the input; however, we rely on their outcomes (e.g., rewards) to compute the target for the Q-function under $\pi_{new}$. Because the probability of sampling these actions differs under $\pi_{new}$, importance sampling is necessary, as outlined in Section 3.1 in our paper and the RL book [1] (Chapter 7.3, n-step Off-policy Learning).

In contrast, when predicting the value $Q(s_t, a_t, a_{t+1}, …, a_{t+N})$ of an action sequence in TOP-ERL, we directly provide these actions $a_t, a_{t+1}, …, a_{t+N}$ as input to the critic transformer. Here, these future actions are deterministic for the Q-function rather than being sampled from a policy, eliminating the need for importance sampling to address varying sampling probabilities.

Can transformer attend more than one state?

It is not completely clear to me if the transformers can attend only the current observation s^k_0 or also the previous ones during training?

In TOP-ERL, we use only a single state $s^k_0$ in the transformer, as we assume the tasks follow an MDP structure, where the task and environment status can be fully represented by a single observed state. Consequently, it is unnecessary to input more than one state.

However, if the task is partially observable, we believe our method could incorporate multiple observations in the transformer critic to infer the underlying system and environment state, potentially improve performance, as discussed in the future work section. Additionally, serveral works in the literature have highlighted the benefits of using transformers in POMDP and scenarios that need memories, such as the work in [2].

Role played by MP?

I am wondering how important are the probabilistic MP in the performance, what if the policy was also a transformer?

MP approaches, as parameterized trajectory generators, have been widely used in fields such as robot learning and path planning [3-8]. The primary advantage of MP is its ability to generate smooth, temporally- and DoF-correlated trajectories, which has been shown to significantly enhance task performance, as evidenced by [6] (one of the baseline methods in our experiment). Additionally, MP functions as a macro-scale skill representer, predicting the entire action sequence (100-500 steps in our experiments) in a single pass.

Directly using transformers as policies is indeed a growing research area, though it remains mostly limited to imitation learning and offline RL, where they are used to predict action chunks and regress behaviors from fixed datasets, typically spanning 3-20 steps.

(To be continued ...)

Dear Reviewer bGrR,

We are grateful for the time and effort you have dedicated to reviewing our submission. In response to your valuable feedback, we have revised our paper and have addressed your concerns to the best of our knowledge, as detailed below.

Update Log [Nov. 18]

• Address the concerns of all reviewers.

• Upload additional experiment and ablation study to Supplementary Material.

  Link: https://openreview.net/attachment?id=N4NhVN30ph&name=supplementary_material

• Revise the manuscript, with modifications marked in blue.

  Link: https://openreview.net/pdf?id=N4NhVN30ph

=======================================================

Revise to identify the novelty

In the current writing, it is hard to identify the novel parts that are proposed by the author and parts that are kept from other works.

We apologize for any lack of clarity in our initial writing. We have revised the paper in Section 4, by explicitly distinguish the contribution made by our model (Section 4.2-4.4) and the techniques adopted from the literature (Section 4.1 and Section4.5).

More details for policy training in section 4.4?

Section 4.4 seems very important to understand how to adapt SAC to episodic reinforcement learning but it seems not clear.

We apologize for the insufficient presentation in our initial submission. We have expanded this section with additional details and included it in the supplementary material Section R.2.

Discussion on the Relationship to Reward Prediction Methods?

Given the resemblance of the critic training to the prediction of rewards that can be used for more effective credit assignment compared to dense and sparse rewards, additional ablation studies or baselines could be discussed.

We thank the reviewer for this insightful question. Our methodology extends SAC [1] by employing N-step TD error for critic training, which estimates the expected return for a given action sequence. Unlike approaches that explicitly predict future rewards, such as in model-based RL, our method avoids constructing a reward or transition model, which can introduce biases and instability, particularly in robotic tasks with complex dynamics. Instead, N-step TD error facilitates effective credit assignment by capturing the cumulative impact of actions over multiple steps.

While model-based methods or reward prediction networks can enhance credit assignment, they require additional modeling complexity and may suffer from inaccuracies in the learned models. Our approach benefits from the simplicity and stability of TD-based updates, seamlessly integrating into the SAC framework. We consider exploring comparisons with model-based or reward prediction-based methods in robotic tasks to be an interesting direction for future work.

Future return after N-step?

In Eq. (8), the future return after N-step is modeled by the critic network during training? It is interesting that if this is the case, the training is stable.

Yes, the future return after N steps is predicted by the target critic network, which is a delayed version of the critic network being optimized. Additionally, we use a V-function $V_{\phi_{tar}}$ instead of a Q-function $Q_{\phi_{tar}}$ to bootstrap the future return after N-steps. This approach is computationally cheaper, as it requires no action sequence input, while still achieving comparably good performance.

Notably, using a V-function to bootstrap the future return aligns with the original SAC approach, as shown in Eq. (2) of [1], where a V-function is used to bootstrap the one-step future return. However, in this early SAC version, the V- and Q-functions were trained independently with separate neural networks. Due to the complexity of training two networks, this version is updated to the version using a Q-function for bootstraping. In contrast, our method employs a shared transformer architecture and parameters for both the V- and Q-functions and do not explicitly train two networks.

More discussion on initial condition enforcement?

In the reviewer’s view, it is hard to grasp the need for the enforcement of the initial condition in section 4.3.1 as s_0^k is already a condition to generate the trajectory in the policy. Some additional intuition about how this is related to the division in segments, or because of the variable length used, or the sampling of trajectories would clarify this part to the reader.

We apologize for the unclear explanation. In ERL, the policy typically predicts an entire action trajectory based on the initial task description, i.e. at time $t=0$. Our method, TOP-ERL, is an off-policy approach, meaning the current policy is updated using data collected from a behavior policy that interacted with the environment, which we refer to as the old policy.

(To be continued...)

Dear reivewer gDNi,

Thank you for your valuable and insightful feedback on our work. We have addressed your concerns to the best of our knowledge.

Update Log [Nov. 18]

• Address the concerns of all reviewers.

• Upload additional experiment and ablation study to Supplementary Material.

  Link: https://openreview.net/attachment?id=N4NhVN30ph&name=supplementary_material

• Revise the manuscript, with modifications marked in blue.

  Link: https://openreview.net/pdf?id=N4NhVN30ph

=====================================================

First off-policy ERL paper?

The paper claims to be "the first off-policy episodic reinforcement learning algorithm," but this seems to be an overclaim. To my knowledge, there are already several related works, such as: Liang D, Zhang Y, Liu Y. "Episodic Reinforcement Learning with Expanded State-reward Space." arXiv preprint arXiv:2401.10516, 2024.

We thank the reviewer for pointing this out. We believe the question arises from the similarities between two distinct concepts: “episodic reinforcement learning” (ERL) and “episodic control-based reinforcement learning” (ECRL).

In our work, we refer to episodic reinforcement learning as defined in foundational RL works such as Episodic REINFORCE [1], Episodic Natural Actor Critic [2], and Episodic Policy Learning [3], which has been further developed in recent studies [4, 5, 6, 7, 8]. This line of research frames the RL problem as optimizing controller parameters to maximize cumulative return across each episode. To the best of our knowledge, TOP-ERL is the first off-policy algorithm developed within this framework.

In contrast, the work by Liang et al. [9], referenced by the reviewer, aligns with a different line of research rooted in Episodic Control (EC) [10, 11]. ECRL typically incorporates a specialized episodic memory to regularize Q-function learning by reusing trajectories with similar state-action pairs (as discussed in Section 2.3 of [9]).

We plan to include a discussion in a future version of our paper to clarify the distinctions between these two concepts and address potential misunderstandings.

Concerns of novelty

The methodology seems to lack novelty, as important components follow the design of previous episodic RL methods. I did not observe other core contributions aside from the reported strong performance. In the experimental section, it would be beneficial to see experiments and analyses related to the gaussian policy and movement primitives module.

Thank you for your feedback. We have revised several parts of Section 4 to explicitly differentiate the contributions of our model from the techniques adopted from the literature.

The core contributions of our approach are summarized as follows: (1) we introduce a method integrating a Transformer as the critic within model-free, online RL framework, (2) we propose using N-step return as the target for the Transformer critic, and (3) we present TOP-ERL as the first Off-Policy ERL algorithm. Each of these contributions is novel compared to previous ERL methods.

The benefits of using Movement Primitives instead of standard Gaussian policy have already been well discussed in previous ERL works, including generating smoother trajectories [4] and capturing the temporal correlations between actions [5]. We briefly discussed these features in the related works (Section 2).

Incomplete Repository?

The code repository is incomplete, lacking the environment configuration files and key training files. Additionally, several important functions, such as ValueFunction, have not been implemented.

We apologize for the missing configs. These configs included personal working directories and HPC usernames, so we initially excluded them to comply with double-blind review rules. We have now cleaned and uploaded them (four configs with the keyword “hpc”) to the anonymous GitHub repository.

The implementations of the value functions were included in the initial code base at https://github.com/toperliclr2025/TOP_ERL/tree/main, with locations listed as below:

• The transformer backbone underlying the critic is implemented in util_nanogpt.py (https://github.com/toperliclr2025/TOP_ERL/blob/main/mprl/util/util_nanogpt.py), adapted from the open-source NanoGPT library (https://github.com/karpathy/nanoGPT)
• The critic class, including components for various configurations such as double Q-networks and target networks, is in seq_critic.py (https://github.com/toperliclr2025/TOP_ERL/blob/main/mprl/rl/critic/seq_critic.py).
• The RL agent, including the N-step return (Eq. 8 in the paper), is in seq_agent.py (https://github.com/toperliclr2025/TOP_ERL/blob/main/mprl/rl/agent/seq_agent.py), with the N-step return implementation on line 304. The variant using the Q-function as the future return is on line 585.

(To be contined...)

Dear Reviewer gDNi,

As the rebuttal period comes to an end, we would like to know if our responses have adequately addressed your concerns. Your feedback will help us make a more informed decision about our submission. Thank you in advance for your time and input.

Best,

TOP-ERL Authors