Efficient Cross-Episode Meta-RL

Additional Benchmarks and Baselines

To address the concern regarding benchmarks and baseline comparisons, we have added the following:

1. New Benchmarks:

   • We included locomotion tasks such as HalfCheetahVel, HalfCheetahDir, and AntDir to evaluate the generalization and meta-training capabilities of our approach.
   • Figure 4 shows the comparison in terms of the generalization capabilities at test time, while Figure 12 presents the meta-training curves for these new benchmarks. Our method, together with TrMRL and VariBAD shows a robust performance across the three tasks.

2. New Baseline:

• We included AMAGO as an additional baseline for the ML10 and ML45 benchmarks in Figure 6.
• This allows for direct comparisons with a strong transformer-based Meta-RL method capable of processing long sequences of transitions across episodes.
• These additions enable a broader evaluation of our approach across diverse benchmarks and a rigorous comparison against a competitive state-of-the-art baseline.
• Results show that ECET achieves superior performance compared to AMAGO, highlighting the effectiveness of our disentangled transformer architecture in capturing task representations efficiently.

Ablation Study

We recognize the importance of an ablation study to isolate the contributions of our transformer architecture (ECET) and its components. In fact, we already provide an extensive ablation study in Section A.6.5 of the appendix, where we systematically analyze the contributions of individual components. We have also added a more comprehensive discussion of the ablations in the main paper in Section 5.3. For the reader's convenience, this study is summarized as follows:

1. Impact of Key Hyperparameters:

• We varied the number of episodes ($E$) in memory and the sequence length ($T$) of sampled transitions, as shown in Figure 28.
• Results show that our chosen configuration of $E=25$ and $T=5$ balances fast convergence and high performance, while extreme values (e.g., $E=50$, $T=50$) result in suboptimal or slower improvement due to inefficiencies in representation learning.

2. Sampling Strategy:

• We compared sampling one transition sequence per episode (our default) against sampling multiple sequences with replacement across episodes in Figure 29 (top plot).
• Results indicate that sampling from episodes individually without replacement leads to faster and more reliable improvements, particularly for smaller $E$ and $T$ values.

3. Policy Input Augmentation:

• We analyzed the effect of augmenting the policy $\pi$ with a linear transformation of the state versus relying solely on the transformer’s output in Figure 29 (middle plot).
• Results show a significant improvement when including the linear transformation, supporting its inclusion in our method.

4. Architectural Variations:

• We ablated the positional encoding in the Cross-Episode Transformer (CET) and simplified our model to a flat architecture similar to TrMRL, as shown in Figure 26.
• Positional encoding in CET leads to suboptimal convergence due to misleading ordering in representations. The flat architecture struggles to capture comprehensive task representations, highlighting the effectiveness of our hierarchical approach.

5.Comparison with Extended Context TrMRL:

• We extended TrMRL to process sequences of the same length as ECET ($E \times T$) in Figure 27.
• Results show that TrMRL with extended context faces significant inefficiencies in extracting task meta-features, highlighting the advantages of ECET's hierarchical design for encoding shorter sequences across episodes.

Clarification of Hypotheses and Claims

1. Hypothesis 1 (Generalization Capabilities):

• The robust test performance in the newly added MuJoCo benchmark in Figure 4 further strengthens our claim that ECET improves generalization.
• Ablation studies further demonstrate that our context sampling strategy and hierarchical architecture are critical to this improvement.

2. Hypothesis 3 (Efficiency and Robustness):

• Comparisons with TrMRL and TrMRL (extended context) in Figure 27 show that ECET achieves superior meta-training efficiency and anytime performance, even when processing equivalent context lengths.
• Ablations of architectural components confirm that ECET’s disentangled transformer design balances computational efficiency with robust task representation learning.

We hope these changes and clarifications address your concerns and demonstrate the robustness of our study. Thank you again for your insightful comments, which have significantly strengthened our work.

We appreciate the reviewer's inquiries and have taken steps to address them thoroughly in our updated manuscript. We have uploaded our updated manuscript which incorporates your feedback.

TrMRL Results
The TrMRL paper focuses on meta-training tasks (e.g., "MetaWorld-ML45-Train"), where the tasks are already seen during training. Citing from Section 5.2 in their paper:

"Finally, for ‘MetaWorld-ML45-Train’ (a simplified version of the ML45 benchmark where we evaluate on the training tasks), the more complex environment in this work, TrMRL achieved the best learning performance among the methods, supporting that the proposed method can generalize better across very different tasks in comparison with other methods."

This setup inherently results in higher performance.
Our evaluation emphasizes meta-testing tasks, which assess generalization to unseen environments. This aligns with the core focus of our work—generalization beyond the training tasks.

When we compare our TrMRL results specifically in the meta-training context, they are consistent with those reported in the TrMRL paper.

Regarding the positioning of our approach

Thank you for pointing out the misrepresentation in our original phrasing regarding prior work that encodes transitions from multiple episodes. We appreciate the opportunity to clarify the novelty and positioning of our approach. Based on your feedback, we have revised the introduction to provide a more accurate positioning of our work relative to existing methods such as VariBAD, TrMRL, and AMAGO. Below, we provide a concise summary of the unique positioning of our approach:

1. Acknowledgment of Prior Work: We recognize and detail the contributions of prior works like VariBAD, TrMRL, and AMAGO in encoding transitions across episodes. This provides an accurate context for understanding the advancements our method offers.
2. Specificity of Our Contribution:

• Unlike VariBAD and RNN-based methods, our transformer architecture effectively handles long-term task dynamics across multiple episodes without being hindered by vanishing gradients or memory constraints.
• Compared to TrMRL and AMAGO, our method achieves a better balance of computational efficiency and performance by reducing runtime complexity while maintaining the ability to process cross-episode information. Further, our hierarchical design using the IET and CET hierarchically enables us to bias the learning to explicitly make use of intra-episodic information. In flat architectures (such as TrMRL) this has to be learned. Depending on the available data, this might be far from trivial to learn, even if it might be beneficial.

3. Practical Relevance: Our work addresses the computational and generalization challenges that have limited the applicability of Meta-RL methods in real-world settings, such as robotics and dynamic environments. Through empirical evaluations, we show that our method achieves faster convergence during meta-training while requiring fewer timesteps to generalize effectively to out-of-distribution tasks. This is especially evident in our comparisons with baselines such as TrMRL and AMAGO, where our approach consistently outperforms them in both training efficiency and final generalization performance. These results underscore the practical utility of our method in applications where computational resources and time are critical constraints, making it a more viable choice for real-world deployment.

We believe these changes clarify the novelty and positioning of our work while highlighting its advantages over existing methods. The updated introduction now accurately conveys our contributions and how they advance the state-of-the-art in Meta-RL.

Q4: Generalization and Novelty of the Architecture

• Conceptual and Empirical Basis for Generalization:

  • Vaswani et al. (2017) highlights the transformer’s ability to model long-range dependencies through self-attention mechanisms, making it effective for sequence modeling tasks. This conceptual advantage, combined with our disentangled intra- and cross-episodic dependencies, supports our method's capacity to generalize.

  • While approaches using multiple transformers in a hierarchy have been proposed (Shang & Ryoo, 2021; Correia & Alexandre, 2022; Mao et al., 2023; Zhang et al., 2023), we are the first to propose a hierarchical architecture that through capturing intra- and cross-episodic meta-features aims to process long sequences more efficiently in terms of runtime complexity:

    • Shang & Ryoo (2021) use hierarchical architectures for offline RL, where the first tier (Step Transformer) transformer encodes a single observation (s, a, r) step, whereas the second tier (Sequence Transformer) encodes the sequence of Step Transformer embeddings.
    • Correia & Alexandre (2022) use hierarchical transformers for offline RL, where the first tier (High Level Mechanism) processes states to generate subgoal embeddings, whereas the second tier (Low Level Mechanism) processes states, actions and subgoals to generate the next action.
    • Mao et al. (2023) use a Vision Transformer in the first tier to encode the observations instead of a CNN, whereas the second tier encodes the sequence of the Vision Transformer embeddings.
    • Zhang et al. (2023) use hierarchical transformers for multi-agent systems. They use the first tier (Inner Transformer Block) to encode individual agent states, whereas the second tier (Outer Transformer Block) encodes the sequence of all agent states and previous actions.

  • These works, as well as TrMRL and AMAGO as transformer-based baselines from our paper, when processing sequences of transitions of length $T$ spanning multiple episodes $E$, have a runtime complexity O$E^{2}\times T^{2}$ whereas ECET, due to its architecture, has a runtime complexity O$E^{2}+T^{2}$.

  • Empirical evidence from our ablation studies (e.g., Figure 26) further demonstrates that our architectural choices improve generalization compared to flat architectures.

• Relation to Task Augmentation Methods:

  • The referenced works (Lee et al., 2021; Rimon et al., 2022) address task augmentation in meta-RL, a different approach from task inference, which is the focus of our method.
    • Lee et al. (2022): Uses task augmentation with imaginary tasks from a latent dynamics mixture to enhance generalization.
    • Rimon et al. (2022): Samples tasks from a KDE to augment the training set and improve the diversity of the task distribution.
  • These methods are complementary to our approach and could be integrated with ECET to further enhance its performance. We have added a discussion of these works in the Related Work section and highlighted task augmentation as a promising direction for future work.

Thank you for your thoughtful feedback and for engaging in discussions to help us improve our work.
We sincerely apologize for any difficulty in tracking our responses due to the lack of explicit numbering. Below, we address your latest comments and the remaining concerns systematically.

Q1: Comparison to VariBAD and RNN-based methods

• TBPTT and Its Trade-offs:
  While TBPTT (Truncated Backpropagation Through Time) can mitigate vanishing gradients and memory constraints in RNNs, it introduces certain trade-offs:
  • TBPTT requires manually defined truncation intervals, which may hinder long-term dependency learning, especially in tasks with sparse rewards or delayed returns.
  • This manual truncation can cause a loss of information critical for tasks requiring far-reaching temporal dependencies.
• Advantages of Transformer Architectures:
  • Vaswani et al.(2017) showed that transformer-based architectures inherently avoid these limitations by processing sequences in parallel, while also being well-suited for capturing long-term task dynamics.
• Empirical Evidence:
  • The key difference between ECET and VariBAD lies in the task encoding mechanism: ECET uses a two-tiered transformer architecture, whereas VariBAD relies on an RNN. Both methods share RL2 as the base meta-RL algorithm.
  • Our experimental results consistently demonstrate that ECET outperforms VariBAD, underscoring the benefits of our approach on efficiency in terms of meta-training timesteps and long-term dependency handling.

Q2: Explicit Use of Intra-Episodic Information

• Architectural Rationale:
  • Our disentangled architecture explicitly separates intra-episodic and cross-episodic representations using two distinct transformer components:
    • IET (Intra-Episodic Transformer): Captures task-specific behaviors within an episode.
    • CET (Cross-Episodic Transformer): Aggregates meta-level task characteristics across episodes.
• Empirical Justification:
  • As shown in our ablation studies (Figure 26 in Appendix A.6.5), flattening the architecture (as in TrMRL) to process both types of dependencies in a single transformer leads to poorer generalization. This result validates the architectural choice to disentangle these processes.

Q3: Latency/FLOP Comparison and Practical Utility

• Focus on Meta-Training Efficiency in Terms of Timesteps:
  • Our claims of practical utility focus on the efficiency gains in meta-training timesteps achieved by the disentangled transformer architecture. This stands in contrast to approaches that rely on a single transformer layer to process long sequences spanning multiple episodes. Importantly, we did not intend to overclaim practical applicability. Our primary claims relate to the performance comparison of ECET with TrMRL and AMAGO, as demonstrated in the meta-RL setting.
• Clarification of Claims:
  • We acknowledge that the statement regarding the practical utility of ECET, as presented in the rebuttal, may have been interpreted as an overclaim. To address this, we have already revised the corresponding statement in the rebuttal response to all reviewers to ensure clarity.
  • Importantly, this claim does not appear in our manuscript, as we have not made any assertions about runtime or FLOP superiority compared to RNN methods there. To further reinforce this point, we have now explicitly stated in the introduction of the manuscript that our primary claims relate specifically to meta-training timestep efficiency, as demonstrated in our results. That said, we do have superiority in terms of runtime complexity compared to the other transformer-based baselines, and we clarify that in the end of Section 4 in our manuscript.

Thank you for your feedback and for raising these important concerns. We value the opportunity to clarify the novelty and contributions of our work, particularly in relation to TrMRL (Melo, 2022) and its memory system, as well as the broader impact of our approach.

Novelty and Central Contribution

While our approach indeed builds on prior work, we believe it introduces the following central contributions that go beyond incremental improvements:

1. Disentangled Transformer Architecture:

   • Our method employs a hierarchical transformer-based architecture (ECET) that disentangles intra- and inter-episode representations. Specifically:
     • Intra-Episode Transformer (IET): Encodes short sequences of transitions within individual episodes.
     • Cross-Episode Transformer (CET): Aggregates these intra-episode representations across multiple episodes to capture global task characteristics.
   • This disentanglement enables ECET to handle both short-term dynamics and long-term cross-episode task patterns more effectively than TrMRL, which uses a monolithic transformer that processes sequences without distinguishing between intra- and inter-episode information.

2. Efficiency in Handling Context:

   • TrMRL's episodic memory system uses a single transformer to process sequences, limiting its capacity to scale to longer sequences due to runtime complexity. Our hierarchical design allows for shorter sequences to be processed in IET before aggregation in CET, reducing computational overhead and improving training efficiency in terms of timesteps.
   • This advantage is particularly evident in out-of-distribution (OOD) tasks, as our method generalizes more effectively to unseen task variations while maintaining computational efficiency (Figures 4, 5, 6, 7).

3. A Broader Scope for Cross-Episode Experiences:

   • Unlike TrMRL, ECET explicitly integrates information from multiple episodes to build a more comprehensive task representation. This approach is critical for tasks requiring adaptation to nuanced or complex inter-episode relationships. In other words, TrMRL is well poised to identify the task dynamics, which can help to adapt the policy to improved performance. However, by only looking at the dynamics within a single episode TrMRL is unlikely to capture how the previous policies rollout relates to the updated policies rollout. Thus TrMRL will not be able to exploit previous policy behavior in areas where it was good or avoid it explicitly where it was bad. Our approach on the other hand can identify the task as well as how the policy/policies differ across rollouts. Overall we would argue that our approach thus gets us much closer to in context learning to learn by reinforcement as we capture environment dynamics, coupled with learning dynamics.

Generalization Beyond Similar Tasks

We agree that generalization to tasks completely different from the meta-training set remains challenging. However, we emphasize that our method outperforms state-of-the-art baselines, including TrMRL and AMAGO (see also response to Reviewer ixfQ), on out-of-distribution tasks. While our approach leverages similarities across tasks (as stated in Hypothesis 3), this is an inherent limitation of most meta-learning methods, including TrMRL. Importantly:

• Our method demonstrates improved performance across timesteps, faster meta-training convergence and improved meta-test performance, enabling agents to generalize to tasks with greater variations.
• The inclusion of diverse benchmarks (Figures 4, 5, 6, 7) showcases our method’s robustness across task distributions with varying degrees of similarity to the meta-training set.

We acknowledge this limitation and view it as a promising direction for future work to extend ECET’s capabilities to handle tasks with minimal or no similarities to the training distribution.

Thank you for raising these insightful points and for recognizing the challenges inherent in experimentally substantiating claims about task feature generality. We appreciate the opportunity to elaborate on our thoughts regarding these aspects.

Definition and Measurement of Task Feature Generality

The concept of "generality" could benefit from a more nuanced consideration. While it's typically framed as the ability of task features to encode transferable task-relevant information, here is a deeper challenge: these features must also help inform how learning progresses, not just differentiate between tasks as we are following the RL2 paradigm. This introduces algorithm dependence into the notion of generality since the task features are tied to the algorithm used to generate the embeddings or data. As such, disentangling the contributions of task-specific features from those tied to algorithmic behavior is a significant challenge. Developing metrics that isolate and measure this pure notion of generality without conflating algorithmic influence might be particularly difficult and is worth discussing further.

Confounders in Embedding Clustering

We acknowledge that clustering embeddings for some tasks, as presented in Figure 13, may be influenced by factors beyond task feature generality. Potential confounders including limited data or noise in the sampled sequences that are used to generate task representations could lead to suboptimal clustering behavior.

Challenges in Systematic Experimentation
Concrete examples could clarify the challenges. For instance:

1. Curating a Diverse Task Set: Imagine generating task embeddings using multiple instantiations of our learning algorithm (i.e., with different hyperparameter configurations) across tasks with distinct dynamics (e.g., gridworlds, continuous control, and adversarial games). Ensuring diversity while maintaining meaningful comparisons across these two dimensions of variability would require careful and labor-intensive curation.

2. Ground Truth for Task Differences: Establishing ground truth becomes even harder when considering multiple configurations of the algorithm used to generate learning trajectories on different tasks. Each configuration might result in subtle differences in task difficulty or structure, which could lead to discrepancies in how tasks are represented. For example, a task embedding generated using an instantiation with one hyperparameter setting that focuses on exploration might appear significantly different when the algorithm is reconfigured and less exploratory, even if the underlying task dynamics remain the same. Developing similarity metrics that remain robust across such algorithmic variability adds a layer of complexity to the already challenging task of defining task differences.

3. Scalability: Evaluating embeddings across tasks, and hyperparameter configurations could become prohibitively resource-intensive. This computational demand would grow exponentially if the diversity of tasks or algorithms increases, making systematic experimentation at scale a significant challenge.

Despite these challenges, we believe that such systematic experimentation is an important direction for future research and could yield valuable insights into the nature of general task features.

We appreciate your recognition of the promise shown by our results concerning Hypothesis 1. While our current work provides initial evidence for the hypothesis, we agree that additional logical steps, such as systematically linking task similarities to embedding distances, would significantly strengthen the claim. We see this as an exciting avenue for future research and thank you for your thoughtful suggestions, which will help guide this exploration.

Thank you for your question.

In Figure 6, the "Meta-Learning Timesteps" on the x-axis represents the total number of individual timesteps accumulated across all meta-training tasks during the meta-learning process.

Regarding the evaluation of the models during meta-test(adaptation), they were assessed using 25 episodes per task for MetaWorld and ManiSkill, and 5 episodes for the MuJoCo benchmark. This ensures a consistent and fair comparison of their performance in adapting to new tasks.

We hope this clarifies your question and are happy to provide further details if needed.

I thank the authors for thoughtful rebuttal and additional experiments they provide. Before I revise my score, I have a question to ask: Figure 6 reports "Meta-Learning Timesteps" on x-axis. Is that the amount of iterations of gradient updates during training? If so, how many environment episodes were used to evaluate the models for the adaptation?

Thank you for your constructive feedback. We have incorporated your suggestions and made significant improvements to our study by adding new benchmarks, including AMAGO as a baseline, and clarifying the implementation details of TrMRL. Below, we provide a detailed account of the changes made to address your concerns.

On the Implementation of TrMRL

We appreciate your point regarding the architectural differences between TrMRL and our method. To ensure a fair comparison:

• We used the publicly available TrMRL implementation and followed the exact pipeline proposed by its authors, as described in their code and paper.
• TrMRL utilizes a transformer in the inner loop of the RL² algorithm. Our implementation makes no modifications to this setup.
• Hyperparameter Settings: -The authors of TrMRL provide the tuned hyperparameters for the MetaWorld and MuJoCo benchmarks, in the TrMRL repository.
  • For Maniskill, we applied the same settings as in MetaWorld due to the absence of explicit guidance in their repository.
  • For our method, we also used the exact same hyperparameter settings for MetaWorld and ManiSkill, detailed in Table 1 (Appendix A.3). The only difference in hyperparameters comes from hyperparameters that are introduced for incorporating cross-episodic information.

On Comparing to AMAGO

Thank you for suggesting AMAGO as a baseline. We have now included AMAGO in our experiments and evaluated its performance on ML10 and ML45 benchmarks:

• Results are presented in Figure 6 and other figures summarizing ML10 and ML45 outcomes.
• Key Findings:
  • Transformer-based methods outperform non-transformer approaches.
  • ECET achieves superior performance compared to AMAGO with at least twice as high a success rate in ML10 and ML45 each, highlighting the effectiveness of our disentangled transformer architecture in capturing task representations efficiently.

By including AMAGO as a baseline, we broaden the scope of our evaluation and ensure that our results are positioned rigorously within the state-of-the-art.

Additional Benchmarks

To further address concerns about benchmarks and baseline comparisons, we have added the following:

• We included locomotion tasks such as HalfCheetahVel, HalfCheetahDir, and AntDir to evaluate the generalization and meta-training capabilities of our approach.
• Figure 4 shows the comparison in terms of the generalization capabilities at test time, while Figure 12 presents the meta-training curves for these new benchmarks. Our method, together with TrMRL and VariBAD shows a robust performance across the three tasks.

Conclusion

We are grateful for your thoughtful feedback, which has significantly strengthened our manuscript. By including new benchmarks, adding AMAGO as a baseline, and clarifying our experimental setup, we provide a more robust evaluation of ECET. These revisions ensure that our claims of SOTA performance are rigorously supported and transparently presented.

Thank you again for your insightful comments, which have greatly contributed to improving the clarity and depth of our work.

I understand the authors' position on the matter of hyperparameter optimization, and I do not question their academic integrity. As I understand it, one of the main contributions of the paper is claiming SOTA results; therefore, I believe it is vital to eliminate any ambiguity regarding the evaluation protocol. It may be that authors of, say, VariBAD or AMAGO did not exhaustively tune their hyperparameters, and one of the baseline methods might actually underperform because of it. I am quite pedantic about this matter, so I stand by my score.

At the same time, I admire the authors' commitment to the rebuttal and their willingness to answer my questions. Thank you.

We appreciate your thoughtful feedback and thank you for your willingness to reconsider the score. Below, we address your concerns in detail:

1. Hyperparameter Optimization
   We understand the importance of conducting a thorough hyperparameter optimization for all methods in comparative studies. However, as mentioned, our approach aligns with the hyperparameter settings used for TrMRL, as detailed in Table 1 in Section A.3 of the Appendix, with the exception of architecture-specific configurations for ECET. To clarify, we did not perform any extensive hyperparameter optimization of our method. While we acknowledge the importance of hyperparameter optimization (HPO), the resource requirements for meaningful searches across all methods are substantial. Nevertheless, we believe that this constraint does not undermine the key insights and contributions of our work, which focus on the efficiency and performance gains of ECET in a meta-RL setting.

2. Comparison with AMAGO-2
   We sincerely thank you for bringing the revised AMAGO-2 paper to our attention. While we acknowledge its advancements in Multi-Task RL, it is important to note that its primary focus aligns more closely with Hypothesis 2 of our work. Specifically, AMAGO-2 primarily addresses generalization to parametric variations of tasks (e.g. more difficult levels) which exist in the meta-training set. This is also reflected in the fact that they don't test the performance of their approach in the meta-test set of the ML45 benchmark, instead they only show evaluation performance on ML45 meta-training tasks in Figure 4. This contrasts with our focus on broader generalization in meta-RL, including adaptation to entirely new tasks in the meta-test set.
   Furthermore, the results reported for AMAGO-2 for ML45 (~0.6 Success Rate at 5e7 meta-training timesteps) are comparable to those we obtained with the original AMAGO. Based on our results with AMAGO, and Figure 4 from AMAGO-2, ECET consistently outperforms both in terms of performance on the meta-training set and efficiency at the same budget of timesteps. Thus, while AMAGO-2 makes valuable contributions to Multi-Task RL and generalization to parametric variations of tasks, its approach and problem formulation differ from those of our work. However, we do see it fitting to add it to our related works, and we have revised Section 2 of our manuscript accordingly.

3. Practical Utility for Robotics
   We appreciate your comments regarding the applicability of transformers in robotics. We agree that deploying transformers in robotic systems poses significant challenges, as highlighted in [1]. However, our claims regarding the practical utility of ECET focus on the efficiency gains offered by the disentangled architecture we propose, which processes context windows more effectively. This stands in contrast to approaches that rely on a single transformer layer to process long sequences spanning multiple episodes. Importantly, we did not intend to overclaim practical applicability to robotics. Our primary claims relate to the performance comparison of ECET with TrMRL and AMAGO, as demonstrated in the meta-RL setting. To ensure clarity, we have amended the wording in our reply to all reviewers to prevent any potential misinterpretation of these claims. We appreciate your feedback on this and have taken steps to address it.

We hope this addresses your concerns and provides clarity on the contributions and scope of our work. We remain grateful for your feedback and are open to further discussion.

[1] Schmied, T., Adler, T., Patil, V., Beck, M., Pöppel, K., Brandstetter, J., ... & Hochreiter, S. (2024). A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks. arXiv preprint arXiv:2410.22391.

I am leaning to change my score to 6 since authors addressed most of my questions, however I still have some concerns that, I believe, should be taken into account:

1. To my mind, SOTA results should be backed up with a serious hyperparameter search of all the methods authors choose to compare to, not only the proposed one. However, I understand that it requires a large amount of resources. This is the main reason I do not revise my score above 6.

2. After the submission period of ICLR"25 had ended, there appeared a revised version of AMAGO-2 [1], which reported similar or superior results in Meta-RL setting and used more complicated environments to evaluate their method. At the same time, I know this paper should not be considered as a prior work during the review, but I still feel obliged to mention its existence here.

3. Comparisons with TrMRL and AMAGO consistently show ECET’s superiority in both efficiency in terms of timesteps and final performance, highlighting its practical utility for resource-constrained real-world applications like robotics and dynamic environments.

   Using transformers is a known challenge for the robotics. As mentioned in [2], a typical sampling rate of a robotic controller varies from 100Hz-1000Hz, which means 15K context length for an episode of 15 seconds. Following this, I would really refrain from such overclaims of practical utility of the proposed method without extensive evaluation first.

[1] Grigsby, J., Sasek, J., Parajuli, S., Adebi, D., Zhang, A., & Zhu, Y. (2024). AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers. arXiv preprint arXiv:2411.11188.

[2] Schmied, T., Adler, T., Patil, V., Beck, M., Pöppel, K., Brandstetter, J., ... & Hochreiter, S. (2024). A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks. arXiv preprint arXiv:2410.22391.