First-Explore, then Exploit: Meta-Learning Intelligent Exploration

1. Although the idea is intuitively well motivated, it might still be good to have some experimental/theoretical motivations behind the intuition.

2. Minor comments:

a. In Fig.3, there is too much space in the fifth row in the caption.

b. Structure-wise, I feel like section 3 can be splitted into 2 different subsections, so that readers can have a clearer overview of it. For example, two subsections could be “meta-rl” subsection and “other works addressing exploration”. Of course you don’t have to change.

However, the writing of the paper is not ready for publication, which hinders readers from understanding it easily and is its biggest weakness:

• First of all, the paper should be reorganized and have more details of the proposed framework in the main text. The current version only has a vague description of the framework in the main text, which doesn’t provide adequate information for the reader to understand the method. I suggest the authors should at least describe a complete instance of the proposed framework in the main text.
• Second, the paper is full of typos and grammatical/format errors, which create significant barriers for the readers. See below for a few examples before pape 5:

1. In the second paragraph of the Introduction, there should be a comma before “e.g.”
2. In the definition of coincidental exploration, “exploitation” should be “exploration”
3. On page 2, “slow, and” should be “slow and”
4. On page 2, “In the dungeon” should be followed by a comma. Same for “while exploring” and “with the fixed dungeon” on the same page
5. On page 2, “be will” should be “will be”
6. On page 3, “Each training step” should be “In each training step”
7. On page 3, “has memory of” should be “has the memory of”
8. On page 3, “the exploratory episodes number” should be “the exploratory episode number”
9. On page 4, “after every explore providing” should be “after every exploration,providing”
10. On page 4, there are $\pi_{\text{explore},\theta,c}$, $\pi_{\text{explore},\theta|c}$. Be consistent with the notation.
11. On page 4, how is the expected reward of an episode defined? Also, it seems that the reward part is missing in $\mathbb{E}(\tau)$.
12. On page 5, “While an innovation” is not a complete sentence.
13. On page 5, “as replacement” should be “as a replacement”
14. On page 5, $k$ is not defined in “\mu_{{1,\dots,k}}”
15. On page 5, “(Fig. 2 A b)” should be “(Fig. 2 A), b)”
16. On page 5, “(c)” should be “c)”

Other minor suggestions

1. Consider adding a pseudocode of an instance of the proposed algorithm in the main text.
2. Consider breaking down some long sentences to make them easier to understand.

In addition, I have some questions about the experiments. See the Questions below.

The paper’s description of the novel method is extremely lacking in detail, to the point that it is not possible to even sketch pseudocode for their idea from the relevant section. Their analysis of exploration in the present literature, while cogent, lacks serious consideration of the benefits of what they call “coincidental exploration”, especially as pertains to traditional frameworks for exploration centered on regret minimization. The meta-learning aspect of the paper is also somewhat doubtful—given the modest description present in the paper, it appears that the “meta”-RL policy which is learned is in fact simply a pair of transformer-based models, each of which act based upon the context of the trajectories seen in a given “training run”. That is not, to my knowledge, the usual sense of the term “meta-RL”. The paper also fails to conduct fair experiments, running the proposed algorithm for 5x as long as the comparators, without adequate justification. Further, this fact is obscured into the appendix. In my view, this behavior represents an attempt to fraudulently represent the proposed algorithm.

1. Comparisons and environments

The paper claims that the presented method can effectively meta-explore, but I have numerous questions regarding the experiments. How does the full proposed approach perform on standard meta-RL benchmarks [1], and how does it compare to meta-rl algorithms like pearl [1] and RL2 [2] on these environments? Is the sample efficiency better because of enhanced exploration, or is there a tradeoff? The paper proposes separate exploration and exploitation policies for meta-learning, how does this compare to meta-cure which also uses a similar approach [3], on the cheetah/walker/hopper meta-envs from that paper ?

2. Idea formulation

The idea of learning how to collect data/explore in order to get sufficient context to subsequently exploit is very common in previous meta-RL algorithms ([1,2,3]). The approach of having explicitly separate policies for exploration and exploitation has also been studied, showing distinct information-seeking behaviors in the exploration phase [3]. Given this, the main difference of the proposed approach is the use of the transformer architecture, but this choice isn't ablated, so it is unclear if this is really important/effective for RL control tasks.

[1] : Rakelly, Kate, et al. "Efficient off-policy meta-reinforcement learning via probabilistic context variables." [2] : Duan, Yan, et al. "Rl $^ 2$: Fast reinforcement learning via slow reinforcement learning." [3] : Zhang, Jin, et al. "Metacure: Meta reinforcement learning with empowerment-driven exploration."