PAE: Reinforcement Learning from External Knowledge for Efficient Exploration

Comment 3 : What is the difference in wall-clock running time between PAE and the other baseline models?

Response:

Thank you very much for your suggestion. Wall-clock running time is indeed crucial for evaluating algorithm performance. To ensure a fair comparison and exclude interference, all algorithms must run independently on the same configured machines. Due to time constraints, we only report the wall-clock running time of different algorithms on KeyCorrS5R3 in Table 1.

Table 2: Wall-clock running time for PAE and other baseline algorithms on KeyCorrS5R3

As demonstrated in Table 2, our PAE method requires comparable training time to the baselines. This similarity is due to both our PAE and baseline algorithms being built on the TorchBeast implementations, with our method not introducing significant additional computation. Therefore, it does not significantly increase wall-clock running time.

Comment 4 : Would it be possible to extract latent knowledge from a collected dataset in a pre-training stage and replace the template-based knowledge in PAE?

Response:

Yes, It would be. In fact, it aligns with our current approach, as mentioned in Section 4.1, where we utilize a pre-trained language model (i.e., BERT-base) to encode and generate knowledge embeddings. We use template-based knowledge solely because the BabyAI environment only provides this type of knowledge. In our further experiments with the new MiniHack environment, we incorporate real natural language feedback from human players during gameplay as external knowledge, eliminating the need for specific templates. The supplemental experiments show that our PAE approach remains superior. Detailed experimental results and non-template-specific knowledge from MiniHack are presented in Section 5 and Appendix A.2 of the revised version.

Thank you for your constructive comments and suggestions, which have greatly enhanced the quality of our manuscript. We have incorporated them into the revision. Below, we provide a point-by-point response to your comments.

Comment 1 : The PAE model comprises three primary components, each with distinct hyper-parameters. Managing these varied parameters can be challenging in real applications.

Response:

We fully understand your concerns. As mentioned in Appendix A.4.1, our implementation of PAE and the reproduction of the baseline algorithm both utilize TorchBeast. Consequently, most hyperparameters align with those in TorchBeast's version, minimizing the need for extensive tuning. For new modules or structures unique to PAE, we conducted a grid search to determine their hyperparameters. The scope of this search and the final hyperparameters are detailed in Appendix A.3.4. Additionally, we have uploaded our source code in the appendix to facilitate easy reproduction of our work by other researchers.

Comment 2 : Since it depends on special environments and predefined knowledge template, PAE is not generalizable to a broader range of tasks.

Response:

To address your concerns, we conducted additional experiments on MiniHack $1$ to demonstrate that PAE is scalable and does not rely on predefined knowledge templates. MiniHack has a more complex state and action space and also possesses direct access to real natural language feedback (Screenshot of Minihack Language) that human players receive during gameplay. We incorporate this systematic feedback into the PAE as external knowledge, making it more natural and realistic, rather than relying on specific templates. For additional information on the MiniHack task, please refer to Section 5 and Appendix A.2 of the revised version.

We conducted comparisons between PAE and the main baselines across two task types, totaling five environments within the MiniHack. For your convenience, we have included a list of the results below and training curves (Screenshot of Training-curves). For detailed information on the new MiniHack tasks and additional experimental results, please refer to Section 5 and Appendix A.2 in our revised version.

Table 1: Comparison of PAE and baseline methods in MiniHack tasks. (mean ± std, convergence steps)

Each entry in Table 1 consists of two numbers: the first denotes the average extrinsic reward achieved, and the second represents the minimum number of stabilizing steps required to achieve a reward plateau, where further significant increases are no longer observed. The results demonstrate that our PAE approach exhibits good scalability and performs well without relying on predefined knowledge, i.e., without specific templates.

Thanks again for your time and effort in reviewing our paper! As the discussion period is coming to a close, we would like to know if we have resolved your concerns expressed in the original reviews. We remain open to any further feedback and are committed to making additional improvements if needed. If you find that these concerns have been resolved, we would be grateful if you would consider reflecting this in your rating of our paper : )

Comment 5 : In Figure 4, I think there is unfair to compare with most of these baselines, since they do not have or use the language guidance. The L-AMIGo is the only one considering language. Are there any other similar work that the authors can compare? The current comparison is not very strong to highlight your work.

Response:

Thank you very much for your valuable suggestion. We have followed your suggestion and introduced another recent Language-Instructed Algorithm called L-NovelD $3$ for comparison with our PAE method. L-NovelD combines natural language with an intrinsic motivation approach to reward states described in natural language that transitions from low to high novelty. Mu et al. $3$ demonstrated that L-NovelD outperforms several popular intrinsic motivation methods, including NovelD, RND, and others, in the BabyAI tasks. Detailed information about the L-NovelD method can be found in Section 5 and Appendix A.4.2 of the revised version.

We conducted a comparative analysis of the PAE and L-NovelD methods across the six BabyAI environments. The table below and training curves (Screenshot of Training-curves) present results for the added L-NovelD and PAE comparisons. For a comprehensive view of experimental results and training curves, please refer to Section 5 of the revised version.

Table 2: Comparison of PAE and baseline methods (mean ± std, convergence steps)

*Due to very limited time, we have so far completed the adaptation and training of L-NovelD on 6 environments in BabyAI, and the training in the 5 additional MiniHack environments is still running. We will update our full experiments in the camera ready version.

Each entry in Table 1 consists of two numbers: the first denotes the average extrinsic reward achieved, and the second represents the minimum number of stabilizing steps required to achieve a reward plateau, where further significant increases are no longer observed. The results demonstrate that our PAE method still outperforms the L-NovelD method, which also utilizes natural language as an aid.

Ref :

$1$ Alec Radford, et al. "Learning transferable visual models from natural language supervision." ICML2021.

$2$ Rohit Girdhar, et al. "Imagebind: One embedding space to bind them all." CVPR2023.

$3$ Jesse Mu, et al. "Improving Intrinsic Exploration with Language Abstractions." NeurIPS 2022.

We appreciate your positive feedback and constructive comments, which have been greatly helpful in enhancing the quality of our manuscript. Below, we provide a point-by-point response to your comments.

Comment 1 : As authors pointed, the current knowledge relies on the specific template, which lacks naturalness and does not fully use the huge ability of the LLM. It is an obvious weakness for this paper

Response:

We fully understand your concerns. We mainly focus on how external knowledge guides agent exploration and therefore directly adopt the pre-defined knowledge set provided by BabyAI without additional processing.

To address your concerns, we have extended our PAE approach to the more challenging MiniHack task $2$ . In addition to having a more complex state space and action space, MiniHack provides direct access to real natural language feedback (Screenshot of Minihack Language) received by human players during gameplay. We integrate this feedback into PAE as external knowledge, enhancing its naturalism and realism without the need for specific templates. For more information on MiniHack tasks, see Section 5 and Appendix A.1 in the revised version.

We conducted comparisons between PAE and the main baselines across two task types, totaling five environments within the MiniHack. For your convenience, we have included a list of the results below and training curves (Screenshot of Training-curves). For detailed information on the new MiniHack tasks and additional experimental results, please refer to Section 5 and Appendix A.2 in our revised version.

Table 1: Comparison of PAE and baseline methods in MiniHack tasks. (mean ± std, convergence steps)

Each entry in Table 1 consists of two numbers: the first denotes the average extrinsic reward achieved, and the second represents the minimum number of stabilizing steps required to achieve that reward. The results indicate that our PAE method performs superiorly without relying on predefined knowledge (i.e., no specific templates). The results of our complementary experiments have addressed the limitation regarding specific templates, and we have updated the discussion in the revised version.

Comment 2 : The current knowledge needs human labeling following the specific template, which is of labor consumption.

Response:

We apologize for the confusion caused by our unclear presentation. Currently, knowledge in BabyAI is automatically provided by the environment and doesn't require additional human labeling. To address your concerns further, we've added the MiniHack task, which provides real natural language feedback (Screenshot of Minihack Language) received by the agent during gameplay, and no specific template requires human labeling. Extended experiments in MiniHack demonstrate that our PAE can handle non-template-specific knowledge in natural language form. You can find the list of knowledge in MiniHack and the experiment results in Appendix A.2 of the revised version.

Comment 3 : I suggest the definition and description of the alignment loss can be placed in the main text, which I finally find in the supplementary.

Response:

Thank you very much for your valuable suggestion. Following your advice, we have relocated the definition and description of the alignment loss to the main text in our revised manuscript. This adjustment ensures that this crucial information is more readily accessible and appropriately highlighted.

Comment 4 : The authors use the cross-attention as the alignment, how about the current alignment work like CLIP and ImageBind? Do the authors consider to use this alignment?

Response:

This is an interesting and worth exploring question. If we go a little deeper into the details, we can unify the mechanism of alignment in PAE, CLIP, and ImageBind. In PAE, we employ Scaled-CrossAttention for aligning knowledge $**k**$ and state $**s**$ , generating the Attention Map between the Query $Q=W\_Q\cdot **k**$ and the Key $K=W\_K\cdot **s**$ . CLIP and ImageBind achieve cross-modal ( $**m**_1$ and $**m**_2$) alignment by utilizing a similar Key-Query manner, generating an Relation Map between the Query $Q=W\_Q\cdot **m**_1$ and the Key $K=W\_K\cdot **m**_2$ .

The primary distinction among these three methods lies in their use of different normalization techniques.

$\text{PAE}: \quad \quad \quad \text{Attention} = \text{SoftMax}(Q\cdot K^T/ \sqrt{d\_k})$

$\text{CLIP}: \quad \quad \quad \text{Relation} = \text{SoftMax}(\text{norm}(Q)\cdot \text{norm}(K^T)) \quad$

$\text{ImageBind}: \quad \text{Relation} = \text{SoftMax}(Q\cdot K^T/\tau)$

Specifically, PAE normalizes using the dimension of the Key vector, while CLIP normalizes both Key and Query, and ImageBind employs temperature coefficients $\tau$ for normalization.

CLIP and ImageBind may offer a significant advantage as they are pre-trained on large natural image datasets. However, there is a domain gap between natural images and unrealistic virtual game graphics. More importantly, our environments (BabyAI and MiniHack) and various baselines all use vectorized inputs, so CLIP and ImageBind are not suitable for direct porting to our task, which is why we chose the learnable Scaled-CrossAttention alignment mechanism. Thanks to your insights, we have included a discussion on these two approaches in Appendix A.1 of revised version.

Thanks again for your time and effort in reviewing our paper! As the discussion period is coming to a close, we would like to know if we have resolved your concerns expressed in the original reviews. We remain open to any further feedback and are committed to making additional improvements if needed. If you find that these concerns have been resolved, we would be grateful if you would consider reflecting this in your rating of our paper :)

Thank you for your encouraging words and constructive feedback. We appreciate your time spent reviewing our paper and have provided point-by-point responses to your comments below.

Comment 1 : I believe that the main limitation of this paper is that the PAE algorithm relies on a pre-defined knowledge set, although the authors have proactively mentioned this in the limitations section.

Response:

We fully understand your concerns. We mainly focus on how external knowledge guides agent exploration and therefore directly adopt the pre-defined knowledge set provided by BabyAI without additional processing.

To address your concerns, we have applied our PAE approach to the more challenging MiniHack task $1$ (Screenshot of Minihack). In addition to having a more complex state space and action space, MiniHack provides direct access to real natural language feedback (Screenshot of Minihack Language) received by human players during gameplay. We incorporate this system feedback into PAE as external knowledge, which is more natural and realistic, without relying on specific templates. For further details on the MiniHack tasks, please refer to Section 5 and Appendix A.2 of the revised version.

We conducted comparisons between PAE and the main baselines across two task types, totaling five environments within the MiniHack. For your convenience, we have included a list of the results below and training curves (Screenshot of Training-curves). For detailed information on the new MiniHack tasks and additional experimental results, please refer to Section 5 and Appendix A.2 in our revised version.

Table 1: Comparison of PAE and baseline methods in MiniHack tasks. (mean ± std, convergence steps)

Each entry in Table 1 consists of two numbers: the first denotes the average extrinsic reward achieved, and the second represents the minimum number of stabilizing steps required to achieve a reward plateau, where further significant increases are no longer observed. The results indicate that our PAE method performs superiorly without relying on predefined knowledge (i.e., no specific templates). The results of our complementary experiments have addressed the limitation regarding specific templates, and we have updated the discussion in the Appendix A.1.

Comment 2 : Although the authors mention that these methods may have higher training/inference costs, I think that given the current popularity of LLM-based agents, this comparison is essential, and higher training/inference costs are not a compelling reason to avoid it. Otherwise, it is difficult to justify the necessity of pre-defined external knowledge and the rationality of the basic settings of this paper.

Response: This is an interesting and worth exploring question. As you mentioned, LLM-based agents are gaining popularity in solving RL tasks, and very recent studies have shown considerable potential. Thank you for the two related works $1,2$ that provided us with some new insights. As far as we know, there are currently two main categories of LLM-based agents for decision-making: 1. Fine-tuning LLMs using RL for decision-making, and 2. Employing LLMs directly for decision-making as plug-ins. These differ significantly from our approach, which focuses on using external knowledge to enhance RL algorithms. In contrast, recent LLM-based agent studies primarily showcase the abilities of LLMs or expand their capabilities using RL methods. In our research, LLMs are a tool to aid agents in understanding the semantics of knowledge.

We appreciate the new insights you've provided and have tried our best to compare the methods you mentioned with PAE. However, we found that these LLM methods currently only accept textual state inputs, which means that they can currently only be used in customized environments. This limitation is clearly stated in the Abstraction of both articles: 'Using an interactive textual environment designed to study...', 'Experiments involving over 100 games in TextWorld...'.

On the other hand, we spent some time trying to run the open-source code in $1$ (https://github.com/flowersteam/Grounding_LLMs_with_online_RL) by converting the environment we used in BabyAI to textual input. Our new experimental results show that this approach does not converge in the BabyAI Key Corridor task we used (using the open source code, all configurations are kept the same). By carefully reading the original paper, we found that Appendix B.1 (page 20 (Screenshot) in the original paper $1$ mentions that "GFlan-T5 has not found yet any robust strategy for the OpenDoor task (being the hardest as the agent must find the right key and discover that the action "toggle" opens the door) in the relatively short allocated time." This is also consistent with our reproduction results.

While for our task difficulty settings, "OpenDoor" is only one of many subtasks that the agent must complete in the correct order to receive the final reward (see Figures 5, 10, and 11 in our paper), and our PAE approach accomplishes these tasks well. We believe that the capabilities of large language models in unknown exploration tasks might be overestimated, but this remains open for community discussion. Regardless, this question is very inspiring, thought-provoking and discussion-worthy, and we will delve into the strengths and limitations of the LLM-Agent in the discussion section. Thanks for your valuable contribution to the depth of our work.

Summary: After experimental validation and thorough review of related work, we conclude that: 1. The current focus of LLM-based agent approaches differs from our PAE method; the former aims to enhance LLM agent capabilities, while the latter focuses on improving RL agent capabilities. 2. The LLM agent approach does not surpass our PAE method in certain tasks (as evidenced by our BabyAI task tests) and is limited to text input only. We have incorporated these insights , as well as related discussions, into Appendix A.1 of the revised version.

Ref :

$1$ Carta, Thomas, et al. "Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning." In ICML 2023.

$2$ Chen, Liting, et al. "Introspective Tips: Large Language Model for In-Context Decision Making." arXiv preprint arXiv:2305.11598, 2023.

Thank you for your prompt response and also thank you for recognizing our supplementary experiment, we are glad that we have addressed your main concern.

Your puzzle about the MiniHack's messages is actually a good question, a corresponding intuitive discussion may better clarify our key ideas to the readers. So, let's make a detailed explanation for it! (Besides, we also provide some intermediate experiment results to make our explanations more intuitive)

(1) The knowledge presented in Table 3 for BabyAI and in Table 4 for MiniHack can be regarded as target states. For example, in MiniGrid, the message "go to the door" can be interpreted as (in this state), go to the door. In MiniHack, the message "the o is killed!" can be interpreted as (in this state), kill the o!. We can view these system text descriptions as the target states that the Planner requires the Actor to achieve after taking a series of actions. Consequently, the Actor must either explore to try new actions to achieve these goals or extract useful strategies from the past actions (trajectories) towards these goals.

(2) Much of the knowledge in MiniHack is not directly contribute to task completion. Given that MiniHack's knowledge is derived from actual player feedback during gameplay, it naturally includes some irrelevant or emotionally messages, such as 'ouch!' and 'never mind'. A rough estimate is provided here: an agent capable of solving the Lavacross task will encounter approximately 80 messages, of which only 6-10 (8-13%) are necessary for a successful trajectory. This requires the Planner to semantically understand these environment messages, filter and exclude those useless knowledges.

We also provide some intermediate experiment results for better understanding: (screenshot of intermedia experiment). We use "LavaCross-Ring" in MiniHack as an example. Due to the complexity of the MiniHack environment, the Planner adopts a more exploratory curriculum. Initially, it presents exploration-based knowledge, such as you see a uncursed flint stone, to help the Actor get familiar with the environment. At a certain stage, the Planner observes the presence of a ring in the environment you see here a ring, dedicates more effort to manipulating the ring what do you want to put on?, and eventually asks the Actor to reach a state of deciding how to place the ring on the correct finger. which ring finger, right or left? to accomplish the task of escaping the dungeon descend. The Planner's curriculum is summarized as follows: "Explore the environment" -> "Find the ring" -> "Put on the ring" -> "Put on the ring on the ringht finger" -> "Descend (success)".

Consequently, we have included the above additional explanations and discussions in Appendix A.2 to clarify how the knowledge works. We hope our response addresses your confusion.

We appreciate the new insights you've provided and your valuable contribution to the depth of our work! Please let us know if you have any further questions.

If you find that these concerns have been resolved, we would be grateful if you would consider reflecting this in your rating of our paper : )

Thank you very much for recognizing our work!

Regarding your following minor question, a possible intuitive explanation is that: in the early stages of training, the Planner actually does not know what kind of goal would lead the Actor to the ultimate reward. Therefore, it might randomly try different messages collected online previously. For example, you see an uncursed flint stone is actually not necessary for completing the dungeon level in "LavaCross-Ring". However, the Planner finds that these messages help enhance the Actor's exploration from its past proposed trajectories. Thus, it reinforces these goals in the early stages. But as the curriculum progresses and the Actor's abilities improve, the Planner will gradually focus on how to lead the Actor to the completion of the main task.

It's noteworthy that whether it's in the early stages when the Actor takes random-like actions, or later when the Actor is trying to complete the Planner's curriculum, there's a high probability that the Actor may trigger new messages. These messages will be instantly collected by the Planner, and these new messages may be organized for the future curriculum. This forms a positive cycle of learning, exploration, and curriculum customization.

We believe that the question of how to enable agents to gather both diverse and informative knowledge during training is quite an open-discussion topic, and we will conduct more in-depth research on this in the near future. We appreciate the inspiring suggestions you have provided.

Meanwhile, we want to thank you for your thoughtful review and insights into our work, which have significantly improved the quality of our paper. Your time and effort are greatly appreciated!

---Respect and Best Wishes from the authors

Summary of our responses related to your comments:

• We demonstrated on the more difficult MiniHack task that PAE can handle more complex action spaces.

• We demonstrated through supplementary experiments that the PAE method does not rely on specific templates.

• We have clarified and explained the role of external knowledge in PAE in the revised version.

• We clarify the Evaluator's design motivation and demonstrate its strong generalization through supplementary experiments.

Ref :

$1$ Mikayel Samvelyan, et al. "MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research." In NeurIPS 2021.

$2$ Mikael Henaff, et al. "A Study of Global and Episodic Bonuses for Exploration in Contextual MDPs." In ICML 2023.

$3$ Jesse Mu, et al. "Improving Intrinsic Exploration with Language Abstractions." In NeurIPS 2022.

Comment 3 : Current experiments require well-defined sub-tasks as well as detection of sub-task completion. BabyAI tasks are about navigating a grid in a particular way and due to task and world structure, it is easy to detect sub-task completion.

Response:

Thank you very much for your comments. You are correct in understanding that the current experiment involves explicit sub-tasks and the detection of sub-task completion. In fact, most of the subtasks provided in the BabyAI environment do not contribute to the completion of the final task, which, on average, only requires 6 to 12 messages (1-2% of all 652 messages) to complete $3$ . The key to earning the final reward lies in identifying useful subtasks and establishing the sequential relationships between them.

In Figure 5 of our paper, we visually demonstrate how PAE progressively learns this logical relationship, and Figure 10 illustrates the process that agents across various tasks must follow to achieve the final reward. Despite intrinsic motivation methods like AMIGo, L-AMIGo, etc., also receiving cues about sub-tasks and sub-task completion detection, our PAE methods outperform them, as indicated in Figure 4.

Additionally, we claim that sub-tasks, like certain actions triggering specific text messages, are more and more common among games, namely "in-game guidance messages". These text messages, which already embedded in the status information, may not be fully utilized by previous baseline methods, limiting their performance. While in our work PAE, the Planner's role is to automatically align and organize these sub-tasks into its curriculum, aiding the Actor in better assimilating external knowledge and exploring the environment.

Last but not least, our new MiniHack environment includes Skill Acquisition Tasks, distinct from the navigation tasks in BabyAI with less structured text knowledge, yet our PAE excels in grasping the sequential relationship between subtasks to complete the final task, surpassing other baselines. backing its effectiveness.

Comment 4 : This method would need sub-task and knowledge alignment so as to detect if a particular piece of knowledge lead to completion of a particular sub-task. I don’t see how it can work any other way. This is a very hard thing to obtain.

Response:

We fully understand your concerns and apologize for any confusion arising from the lack of clear explanations in our paper. External knowledge indeed contributes to subtask completion detection, but this is not the full and crucial role played by external knowledge. Despite intrinsic motivation methods like AMIGo, L-AMIGo, etc., also receiving cues about sub-tasks and sub-task completion detection, our PAE methods outperform them, as indicated in Figure 4. The core innovation of our paper is how to utilize external knowledge to enhance the agent's exploration capabilities. We introduce a generalized framework, PAE: the Planner utilizes external knowledge for creating a progressive curriculum through feedback and guidance; the Actor aligns its current capabilities with the environment state with external knowledge's assistance, and the Evaluator employs external knowledge to detect subtask completion.

Comment 5 : The evaluator currently is highly customized to BabyAI tasks. How did the authors reach the design for evaluator? What challenges if any, do the authors foresee in making the evaluator more general-purpose?

Response: Thank you very much for this insightful question. The Evaluator's primary insight lies in providing guidance to both the Planner and the Actor through the design of intrinsic rewards. Compared to previous approaches, the Evaluator in PAE introduces two key innovations. Firstly, it leverages external knowledge to detect subtask completion, offering a more natural and comprehensible approach. Secondly, the Evaluator guides both the Planner and the Actor by encouraging the Planner to provide incremental guidance that is neither too easy nor too challenging, while motivating the Actor to complete tasks promptly under the Planner's guidance. This design concept has broad applicability, and the Evaluator's key parameters are adaptive, with a step threshold (t*) that automatically grows as the agent's exploration ability improves, capped at 30% of the environment's maximum step count. To validate the generality of the Evaluator, we tested it in the MiniHack environment, where our PAE approach also exhibited superior performance (Screenshot of Training-curves, Screenshot of Addtional-Results). Detailed experimental results are presented in Section 5 and Appendix A.2 in our revised version.

Thank you very much for your constructive comments and suggestions. We have revised our paper accordingly. Below, we will provide detailed responses to each point.

Comment 1 : BabyAI action space is quite limited (6 actions) and simplistic, so are the tasks since they are 2D grid world tasks.

Response : To address your concerns, we have extended our PAE approach to the more challenging MiniHack tasks $1$ . MiniHack consists of procedurally generated tasks within a roguelike game, offering a richer observation space and up to 75 dimensions of structured and context-sensitive action space, presenting huge challenges. The high complexity of this state-action space makes MiniHack one of the most challenging sparse reward environments to explore $2$ . We conducted comparisons between PAE and the main baselines across two task types,totaling five environments (LavaCross-Ring, LavaCross-Potion, LavaCross-Full, MultiRoom-N4-Monster, River-Monster) within the MiniHack. For ease of comparison, we have included a list of the results below and training curves (Screenshot of Training-curves). For detailed information on the new MiniHack tasks and additional experimental results, please refer to Section 5 and Appendix A.2 in our revised version.

We have also provided ample experimental results to demonstrate the effectiveness and scalability of our work. Particularly, we have shown that our approach can be readily extended to more challenging environments.

Table 1: Comparison of PAE and baseline methods in MiniHack tasks. (mean ± std, convergence steps)

Each entry in Table 1 consists of two numbers: the first denotes the average extrinsic reward achieved, and the second represents the minimum number of stabilizing steps required to achieve a reward plateau, where further significant increases are no longer observed. The results indicate that our PAE method performs superiorly and is easy to be adapted to more challenging environments.

Comment 2.1: As pointed out by authors, current experiments have a knowledge oracle and a very specific template for knowledge peices.

Response :

Thank you very much for your comments. The second reason we conducted experiments on the MiniHack task is that, MiniHack provides natural language knowledge (Screenshot of Minihack Language) which is real-world feedback received by human players or agents during gameplay, rather than specific knowledge template in BabyAI. Extended experiments in MiniHack (Screenshot of Training-curves) demonstrate our PAE's ability to handle non-template-specific knowledge in natural language form and. its effectiveness, versatility, and scalability. A list of the knowledge from MiniHack and the experiment results can be found in Section 5 and Table 4 in the revised version.

We mention these limitations in the manuscript because our research originally focused on how external knowledge guides agent exploration. So, it directly followed the templates provided by BabyAI to express knowledge. The results of our latest complementary experiments have addressed the limitation regarding specific templates, and we have updated more discussions in the revised version accordingly.

Besides, we are very grateful for your suggestions to our work, which have significantly contributed to our exploration of the boundaries of the PAE's capabilities. Thank you very much.

Comment 2.2: Planners can observe the whole world, which may be unrealistic for many real-world set up.

This is actually the default setting of the baseline methods in the BabyAI environment, and we strictly followed this setting for a fair comparison. To address your concern, in our newly introduced MiniHack task, both the Planner and Actor utilize identical partial observation inputs. This uniform setting is also maintained across all other baselines. Experimental results confirm that our PAE method can still consistently outperform other baselines.

Thank you for your prompt response. We are happy to hear that we have addressed your main concerns. For your remaining questions, we will provide point-to-point responses:

How is sub-task completion detected in MiniHacks, and are there any pre-defined sub-tasks?

Response: To be precise, there are no man-made pre-defined sub-tasks in MiniHacks. Actually, we employ a general and easy-to-extend approach to automate the process of the Planner acquiring and using knowledge from the environment, which is using the system text messages in the environment as the source of knowledge. Therefore, the completion of these sub-tasks is entirely detected within the game (automatically determined by the built-in trigger logic of messages designed by the game developers). Also, these text messages are embedded in the input observations for PAE and other baselines, while the superior experimental performance of PAE demonstrates that PAE is the best to effectively utilize these textual knowledge, achieving the fastest convergence speed and the best performance.

In fact, the detection of subtasks is also automatically handled by BabyAI's built-in game logic. The only difference is that, BabyAI's subtasks appear more structured and template-like in their message representation and are closer to what humans would imagen as "instructions".

How does the evaluator generate intrinsic rewards in MiniHacks environment?

Response: For the Evaluator, we used the same hyperparameters and the same evaluating logic (Section 4.3 of in our main paper). This, to some extent, may also justify the scalability of our method. Besides, we must acknowledge that our experiments on Minihack were conducted within very limited time. Theoretically, more detailed hyperparameter tuning could lead to even better performance.

Thank you very much for your contribution in clarifying our ideas. Please be free to let us know if you have any further questions. Besides, if you find that these concerns have been resolved, we would be grateful if you would consider reflecting this in your rating of our paper : ) .