Thank you for reading, engaging with, and critiquing our work. We greatly appreciate the time you have spent, and your feedback has enabled us to improve the paper. We are delighted that you support sharing First-Explore with the NeurIPS community and consider it well-motivated and addressing “a problem worth solving.”

We have addressed your concerns and substantially strengthened the paper. We hope you will consider our improvements and raise your score to further support its acceptance.

Weakness Response:

[The choice of $\rho$ seems arbitrary.]

Setting $\rho=0$ ensures all objects have non-negative rewards, so exploring (visiting new locations) only increases rewards. This setting provides the simplest non-deceptive control, and demonstrates that the baselines (and First-Explore) can handle problems without local optima.

Setting $\rho = -4$ ensures that repeated and consistent exploitation is necessary for exploration to be worthwhile, thus creating local optima. Evaluating in this domain examines First-Explore’s central claim: that these exploration dynamics are deceptive and challenging to existing cumulative-reward meta-RL methods, and that First-Explore overcomes this challenge.

When $\rho = -4$, the expected reward of an object is -1 (as the rewards are distributed U[-4, 2]). Furthermore, the expected reward of an object with positive reward is only 1 (as they are distributed U[0, 2]), and only $\frac{1}{3}$ of objects have positive reward. This means multiple exploitations are required before exploration pays off. If an object is found while exploring, the expected reward is -1. However, optimal exploitation yields an expected $\frac{1}{3}$ reward (by avoiding traps and navigating to positive reward objects). Thus, it takes at least four exploitation episodes for exploration to be worthwhile, as $4 \times \frac{1}{3} - 1 = \frac{1}{3} > 0$.

We will include an explanation of these choices in the paper.

[how would behavior vary with $\rho$?]

Between $\rho=-4$ and $\rho=0$ there will be a transition point between the two results shown in the paper. When $\rho=-4$, the baseline fails catastrophically while First-Explore performs well (see Figure 4 A1). When $\rho=0$, the baselines perform reasonably, although First-Explore still outperforms them (see Figure 4 B1). While exciting (and a promising direction of future work), First-Explore outperforming the baselines in the non-deceptive case is not central to the paper’s claims. Increasing $\rho$ from 0 would not qualitatively affect the results since the environment cannot become more non-deceptive. Decreasing $\rho$ cannot worsen baseline performance since they already fail, but it might eventually harm First-Explore due to the sparsity of rewarding objects. We will add text to the paper noting this, along with supporting plots.

[What about more complex environments?]

We have added a more complex domain, Ray Maze (see the main response to all reviewers for details). This new domain is significantly more complex than the Dark Treasure Room. However, the same pattern of performance holds: First-Explore outperforms all controls, which fail catastrophically due to the deceptive nature of the domain.

[How about environments such as MiniGrid or Minihack for example, which have also been used for Hierarchical Reinforcement Learning?]

Thank you for the suggestion. We do not consider First-Explore an example of Hierarchical Reinforcement Learning, which typically involves high-level and low-level policies (e.g., a policy setting subgoals and another achieving them). First-Explore operates by exploring first, then exploiting, without one policy setting subgoals for another. Therefore, MiniGrid and MiniHack would not test the main scientific hypotheses of this paper.

Question Response:

[Is a more complex dark treasure room possible?]

Indeed it is, as we now show! Please see the main PDF, and our main response for details on the new Ray Maze domain.

Why does the value of 2 work for all the configurations? Shouldn’t this value vary[...]?

There may be confusion about when First-Explore switches from exploring to exploiting. After training the explore and exploit policies, the number of exploration episodes $k$ is determined by evaluating on a new batch of environments. Given $k$, the inference policy explores for the first $k$ episodes (regardless of exploration performance) before switching to the exploit policy. Surprisingly, in deceptive domains, this simple approach outperforms all controls. We will clarify this in the text.

What you describe (the exploit policy deciding on the right amount of exploration steps) is a promising direction of future work that we will add to Section 6.

Why does [First-Explore performance] go down first in Figure 4 [and not for] other methods?

When $\rho = -4$, the environments are such that the expected reward of an unseen location (one not visited in a prior episode) is negative, and thus moving in the first episode leads to negative expected reward. Positive cumulative rewards can be achieved because, over multiple episodes, the agent can exploit any positive rewards it has found while avoiding negative ones. Unfortunately, the other methods fail to learn this strategy and instead learn to mostly stay still (so avoiding negative reward in the first two episodes, but also preventing the significant cumulative reward First-Explore achieves by the end).

The reward goes down for 2 episodes (vs. another number), because $k$ was automatically determined to be 2 on this domain. This setting causes the resulting inference policy to explore for the first 2 episodes (and so seek unseen negative-expected-value locations) before then exploiting for the remaining 8 episodes.

Thank you for reading, engaging with, and critiquing our work. We greatly appreciate the time you have spent doing so, and your feedback has enabled us to strengthen the paper. We are delighted that First-Explore tackles “a highly relevant” problem, that the “proposed solution provides valuable insights,” and that the experimental results are “comprehensive.”

We have addressed your concerns and substantially improved the paper. Furthermore, we believe your current positive comments support a higher score. We hope you will consider our improvements and increase your score by 2 to 3 points. Without this score increase, the paper may not be selected for publication.

Weaknesses Responses

[Figures 1 and 2 are not referenced]

The paper now references Figure 1 and Figure 2.

The training process for the explore and exploit policies is not well explained in Section 4.

Did you mean Section 3.2? Section 4 discusses related work. To address this issue, we have added pseudocode detailing the training process (see main reviewer response for details) and improved the writing in Section 3.2.

The inference policy appears to require exhaustive hyperparameter tuning (k), which limits the method's generalization and makes it challenging to apply to other tasks.

As mentioned in Section 3.2, lines 154-157, “k is not a hyperparameter as, unlike hyperparameters, all policy-weight updates are performed independently of k, precluding the need to train the explore and exploit policies multiple times (which is the majority of the training compute-expenditure).” As illustrated in Figure 2, after training, k is automatically chosen (with minimal compute) via a quick interval search on a validation set. It can thus be quickly updated if the test distribution shifts.

[The compared baselines seem outdated.]

VariBAD and HyperX are state-of-the-art cumulative-reward meta-RL methods. This paper exclusively makes claims on cumulative-reward meta-RL behavior, as opposed to final-episode-reward meta-RL. Similarly, RL$^2$ is still frequently used for cutting edge research (e.g., AdA) and remains an important baseline due to its simplicity and power.

[the results are not significant]

The results have a large effect size, are extremely statistically significant, and fully support the claims of the paper. In short, the results are highly significant in every sense.

The paper’s central thesis is that in deceptive domains existing cumulative-reward meta-RL methods fail, and that First-Explore overcomes this challenge. In the deceptive version of all three domains – the bandits when $\mu_1=0.5$ (Figure 3 A1), the Dark Treasure Room when $\rho=-4$ (Figure 4 A1), and the Ray Maze (see main reviewer response) – the meta-RL controls achieve abysmal performance. In these deceptive domains, First-Explore achieves significantly higher rewards than the best performing meta-RL control: 128.36 vs. 56.12 (> 2x improvement), 1.99 vs. 0.16 (> 10x improvement), 0.46 vs. 0.06 (> 7x improvement) for the Bandit results, Dark Treasure Room results, and Ray Maze results respectively. This large effect size reflects a significant and meaningful difference in agent behavior; the controls have learnt to minimize movement and exploration (Figure 4 B1), while First-Explore explores initially at the cost of reward (Figure 3 B1 and Figure 4 A1). All the results are also extremely statistically significant ($p < 2 \times 10^{-5}$). We shall modify the paper to make the results’ significance clearer.

Question Responses

What is the practical objective of the exploration policy, and how can the informativeness of an episode be defined?

Regarding the practical objective of the exploration policy and the definition of informativeness of an episode: From Section 3.2, “the explore policy π explore is trained to produce episodes that are followed by the exploit policy achieving higher episode returns than those seen so far. These episodes are termed ‘informative.’”

Training the explore policy to produce these ‘informative’ episodes (ones that are followed by the exploit policy achieving higher episode returns than it would have otherwise) is detailed in the pseudocode now added to the paper (see the main response). We will also improve the writing throughout the paper to clarify these issues.

[Is First-Explore efficient?]

Is it more efficient than standard-RL applied to each environment?

As the introduction states, “Meta-RL can potentially [be more sample efficient than standard-RL by] expending a large amount of compute to train a single agent that can then rapidly adapt to new environments, even showcasing human-like sample efficiency.” Once the initial compute expenditure is made then the trained algorithm can be extremely efficient on new (in-distribution) tasks at inference time (i.e., when adapting to new tasks).

Is learning two policies more efficient than learning a single policy?

Learning two policies enables First-Explore to avoid catastrophic failure in the deceptive domains. In this sense, First-Explore is far more efficient than the controls, as they fail to learn good behavior regardless of how long they are trained (e.g., in the hostile Dark Treasure Room Domain, RL2 and VariBAD converge to not moving, and HyperX exhibits the same poor performance regardless of training time, see Figure 5 in Appendix C).

Thank you for your continued feedback. We are delighted you are open to raising your scores to support the paper's publication. We hope our responses below (in this and Part 2) fully address your concerns. Please feel free to reach out if you have further questions.

Your questions:

[How are $\pi_{explore}$ and $\pi_{exploit}$ different, as they both load from $\theta$?]

$\theta$ is a container holding the parameters of both policies. The two policies, $\pi_{explore}$ and $\pi_{exploit}$, differ because, while they share some parameters, they each have unique parameters. In particular, our implementation shares all parameters in the same neural network except for the last layer, see lines 161-162 in Section 3.2.

We will update the pseudocode to clarify that $\theta$ represents a combination of parameters for both $\pi_{explore}$ and $\pi_{exploit}$. A comment will also be added to the load_policies function stating that each policy is constructed using its relevant subset of $\theta$.

Why is l_explore added twice to the loss when r_exploit is greater than or equal to best_r?

To clarify, l_explore and l_exploit are distinct terms in the loss function, and each is added only once under specific conditions related to r_exploit and best_r. Is it possible you mistook l_explore as l_exploit at some point?

Quoting from the main rebuttal,

if r_exploit > best_r:
	loss += l_explore
if r_exploit >= best_r:
	loss += l_exploit

To clarify, l_explore is added to the loss function if the exploit return (r_exploit) exceeds the best exploit return observed so far. This addition encourages the explore policy to produce episodes that enhance the exploit policy's performance.

Similarly, l_exploit is added if the exploit return meets or exceeds the best exploit episode return in the sequence. Here, the loss condition incentivizes the exploit policy to achieve high episode returns. Notably, both conditions check the return of the exploit policy, but add different terms to the overall loss. We will emphasise this detail in the pseudocode.

[Is $k$ chosen automatically?]

After training, the optimal value of $k$ is determined by automatically evaluating each candidate $k$ within the interval $[1,n−1]$ where $n$ is the total number of episodes. To evaluate each of these values of $k$, the associated inference policy (explore $k$ times, then exploit $n-k$ times) is run on a batch of target-distribution environments, and the $k$ that achieves the highest mean cumulative reward is selected. This selected $k$ is then used for inference at test time.

We will rephrase the quoted text to clarify the automatic nature of the $k$ selection process, eliminating any implication of manual tuning. We will also make sure to clarify that this automated selection of the value of $k$ happens after training, and is computationally extremely inexpensive vs. training.

[What is the role of the predictor?]

In our implementation, the agent uses its policy parameters $\theta$ to perform rollouts. These parameters are only periodically updated. A second set of parameters, the predictor's parameters $\phi$ learn an improved policy and are updated every batch of environments. Every $T$ learning updates, the agent policy is set equal to the predictor policy. This setup increases behavioral diversity, and enhances training stability.

We will expand the pseudocode and methods section to clarify the distinct roles of the agent and predictor parameters.

Thank you for your quick and clear response. I have one last question: Does First-Explore belong to the meta-learning framework?

To my knowledge, the meta-learning framework operates on the principle of "learning to learn." For example, one learner acquires knowledge that helps another learner to perform a specific task. Typically, the relationship between these two learners is unidirectional. However, the two policies, $\pi_{explore}$ and $\pi_{exploit}$, in your approach seem to learn from each other, making their relationship bidirectional.

Is my understanding incorrect?

Thank you for addressing my questions. I appreciate the effort put into clarifying my concerns, and based on your responses, I am willing to raise my score to 'weak accept'.

Thank you for reading, engaging with, and critiquing our work. We greatly appreciate your time and feedback, which has significantly improved the paper. We are delighted that you consider First-Explore to be “motivated by an important problem” and “an interesting idea.”

We have addressed your concerns below, and by doing so, we have substantially strengthened the paper. We hope you will consider our improvements and increase your score by 2 to 3 points. Without this score increase, the paper may not be selected for publication.

Weaknesses Response

[What about, for example, MetaCURE and DREAM? These methods separate exploration and exploitation]

DREAM and MetaCURE are final-episode-reward meta-RL methods. In contrast, this paper and algorithm address cumulative-reward Meta-RL (see lines 41, 71, 126, 293). An extensive discussion of MetaCure and DREAM is provided in Appendix F (see Section 2, lines 126-128, 'for the sake of completeness, we discuss final-episode-reward meta-RL and its connection to First-Explore in Appendix F'). Please see that Appendix for the answer to your question.

To make it clearer, we will add text to Section 2 identifying DREAM and MetaCure as final-episode-reward meta-RL methods and their difference to First-Explore.

[More complex environments are needed]

We have added a more complex domain, Ray Maze (see the main response to all reviewers for details). This new domain is significantly more complex than the Dark Treasure Room; however, the same pattern of performance holds: First-Explore outperforms all controls, which fail catastrophically due to the deceptive nature of the domain.

Furthermore, although the tabular environments are less complex than Ray Maze, the tabular results are highly informative. First-Explore identifies a challenge where state-of-the-art methods perform abysmally, often learning to remain stationary. Identifying this issue is a significant contribution of the paper, and is best demonstrated in the simplest environments possible (as is usually the case in science). It is surprising, and worth sharing with the NeurIPS community, how even simple environments can be unsolvable by existing state-of-the-art cumulative-reward meta-RL approaches.

[The methods section needs proper formulas or an algorithm box] [What are the explicit formulas of the training objectives?]

We have now included full pseudocode for the algorithm in the paper (see the main rebuttal response) and will edit the methods section for clarity.

Questions

[Why does First-Explore take so long to run on the tabular environments?]

The hyperparameters and choice of architecture are not optimized for short meta-training durations. The transformer size, for example, prioritizes sufficient model capacity for any task over training efficiency on simpler tasks. It was also motivated by the desire to use a standard implementation (GPT2) of sufficient capacity to learn complex relationships. Similarly, the hyperparameter T (see line 184) enables exceptionally reliable training; however, it also increases meta-training time. If a shorter training time was desired, then these choices could be reconsidered, and future work could focus on optimizing First-Explore for such efficiency.

However, the paper makes no claims on First-Explore’s meta-training efficiency. The paper identifies a new challenge with cumulative-reward meta-RL (the problem of needing to forgo rewards to explore), demonstrates the issue empirically, and provides a new method (First-Explore) that solves the challenge and achieves state-of-the-art results. We will summarize these details in the paper, including the exciting opportunity for future work to focus on substantially improving meta-training time.

How does First-Explore perform on more complex tasks (e.g., Meta-World)?

We have added a new more complex domain - Ray Maze. First-Explore performs excellently in this domain (see the main reply for details).

Meta-World’s environments do not require exploration that sacrifices rewards in early episodes, which is the central challenge this paper addresses. Evaluating First-Explore on such a benchmark would be interesting for future work, but it is beyond the scope of this paper.

How does the proposed method compare with MetaCURE and DREAM?

MetaCURE and DREAM are final-episode-reward meta-RL methods designed for and applicable to the final-episode reward meta-RL setting. This paper exclusively makes claims about cumulative-reward meta-RL dynamics (see lines 41, 71, 126, 293, etc.). While comparing First-Explore to these (or any) methods is interesting, it is not essential to the scientific questions addressed here and is beyond this paper’s scope. That said, we do provide a discussion of this issue in Appendix F (please see it for the most direct answer to your question) and we speculate in the future works section that First-Explore may be applicable to the final-episode-reward setting.

Conclusion

Thank you again for your feedback. We hope you will support a score that would lead to our work being shared with the NeurIPS community.