Policy Learning from Tutorial Books via Understanding, Rehearsing and Introspecting

Q1: It was difficult for me to follow what exactly what does in the method.

We are sorry that the current presentation of URI is not straightforward enough. As URI’s implementation involves interactions among several components from different research domains, it indeed might look complex, especially for readers who are not familiar with one of these domains. In response to this question, we plan to add a pseudo code to the revised paper (see Algorithm 3 in the attached PDF) to make the interactions unambiguous. Besides, we commit to open-source our full-procedure code to help users understand the details of this project. We kindly recommend the reviewer check our open-source plan in the global response. We hope these efforts will make this study more easy to follow.

Q2: A class of methods that use LLMs as the backbone is missing.

Thanks for the valuable suggestion. RT is a classic baseline missing before. We implement an RT-style policy for football based on LLAMA and report the results in Table 2 and Figure 1 in the attached PDF. We use the same 7,500 initial states as URI and the rule-based actions to train the RT policy. After training, the loss can be reduced to normal as shown in Figure 1. However, though we tried our best to improve the performance, RT-policy turned out to fail in reaching the goal (See Table 2). We assume it is due to the size of training data is too small to be used for generalizable policy imitation. We will keep on trying but we cannot guarantee that we can get a better result.

In response to this suggestion, we plan to (1) add RT-branch studies to the related work and (2) add the best performance we can reach to the revised paper.

Q3: It would have been helpful to see how well URI does in a head-to-head match with the baselines.

We agree that the head-to-head evaluation is valuable and will make the effectiveness of URI more convincing. Due to the limitation of the current simulator implementation, it will require considerable engineering work to modify the current Football environment to support the simultaneous evaluation of two policies. We are trying our best to finish such a job and commit to reporting the results in the revised version.

On the other hand, the extended experiment in Tic-Tac-Toe does include head-to-head evaluations in the attached PDF. The results show that URI significantly outperforms the two major baselines LLM-as-agent and LLM-RAG with +66% and +44% net win rate. We hope this result can also solve the reviewer's concern.

Q4: How does this method relate to the use of background textual knowledge in "Motif: Intrinsic Motivation from Artificial Intelligence Feedback" and "Learning to Model the World With Language"?

Thanks the reviewer for pointing out these two related works. We will add a brief introduction and comparison with them in the related work section. Specifically,

1. Motif: Intrinsic Motivation from Artificial Intelligence Feedback: the language is used as a caption of the observation. LLM is used as a surrogate reward model to output an evaluation (preference) of the observation (state) based on its caption. Such a preference model is then used to distill a reward model. Given this reward model, the policy is still trained in an online fashion. This could be classified into the LLM as a reward model category in our related work.
2. Learning to Model the World With Language: the observation contains both image and language. LLM is used as a part of the world model to predict future observations. The training paradigm is still the traditional Model-based RL, but the model here is multi-modal. This could be classified into the "LLM as dynamics model category" in our related work.

Q5: How beneficial is learning with an LLM versus learning a policy with the task reward and no LLM?

Since the PLfB is more like in a offlineRL setting, we tested the performance of CQL as the result of policy learning with the task reward and no LLM. The results are in Table 2 in the attached PDF. In summary, CQL also achieved a competitive performance compared with other baselines, i.e., LLM-as-agent and LLM-RAG, but still significantly underperforms URI (0.07 vs 0.38 in average GDM). This result demonstrates the benefits of URI in improving the policy's performance compared with standard policy learning methods.

Q6: The limitations are not discussed in the main body of the paper.

We would like to kindly point out that the limitation of this paper is mentioned in Section 7. Since the space is limited in the main body, we leave a link to Appendix F in Section 7, where give our full discussions and limitations of this study.

Q1: add baselines: (1) distilled policy from the imaginary dataset generated directly by the GPT without the information from the books; (2) compare it with some other policies learned with conventional RL algorithms

Thanks for the valuable suggestions. In the rebuttal period, we implement the two baselines named “URI w/o BK” and “CQL”, which can be seen in Table 2 in the attached file. Since the PLfB is more like an offline setting, we use a popular offline RL baseline CQL to demonstrate the results of conventional RL algorithms. We found that pure URI without book knowledge and pure offline RL algorithm can reach competitive performance compared with the baselines but URI still achieves significantly better results (0.38 vs 0.04 in average GDM for “URI w/o BK” and 0.38 vs 0.07 for “CQL”). The results demonstrate that purely using the dataset or the inner knowledge of LLMs cannot should the problems. We kindly recommend the reviewer check the detailed results in the attached PDF.

In response to the suggestion, we commit to adding these two baselines to our revised paper. We believe it will make the effectiveness of URI more convincing.

Q2: Why not provide the pseucode for this work in the anonymous repository? Maybe the reproducibility of this work can be further enhanced.

We plan to add a pseudo code to the revised paper (see Algorithm 3 in the attached PDF) to make the interactions among these components unambiguous. Besides, we commit to open-source high-quality code to help users understand the full details of this project. The global response contains the complete plan and scope of the open-sourcing.

Q3: The caption and illustration of Figure 1 are a little bit confusing. Personally, I think book tutorial is also a kind of data & The Figure 2 is somewhat redundant, considering the detailed 3-stage framework: understanding, rehearsing, and introspecting, has been clearly provided in the Figure 3.

Thanks to the reviewer for pointing out these two presentation problems. We agree that the book tutorial can also be regarded as a source of data. We will change it to “interaction trajectories” in the revised version. Figure 2 is the proposed general URI methodology for the problem of PLfB and we expect the readers to quickly develop an idea about our methodology. Figure 3 is more about solution implementation: it provides an overview of the exact implementation for PLfB in this study. We will improve both figures and captions to make such a distinction more clear.

Q4: More datasets & applications can be test.

We acknowledge that applying URI in just single domains is not enough to demonstrate the generalizability of the methodology. To alleviate such a concern, we build a new proof-of-concept benchmark based on the classic Tie Tac Toe game (TTT) and verify URI's performance on it. In short, URI continues to behave well. The benchmark setting and more results are elaborated in the global response

Q1: Prompting Strategy Details & Exploration: Could you elaborate on the specific prompting strategies used to generate the imaginary dataset? How do you believe different strategies might influence the quality and effectiveness of the policy learned?

We agree that the design of prompts matters in LLM-related methods. Though we do not focus on propose new prompting techniques, in our practice, the following principles of prompt designs are important in this projects:

1. Before output the results, let LLMs output the “thoughts/analyze” firstly, and we will give an example of thoughts in the prompts. Reference to the “Response example“ parts in our prompts in Appendix C and D for more details.
2. Make our requirements as explicit as possible. Reference to the “Requirements“ parts in our prompts in Appendix D.
3. As mentioned in Section 5.2, instead of using natural language to represent the knowledge, we use pseudo-code.

Since (1) and (2) are well-known principles and have been verified in other studies many times, in the following, we ablate the effects of knowledge representation. We conduct the same experiment setting as Figure 5(a) in the main body by using natural language to represent the knowledge. As shown in the table below, if we switch to the natural-language-representation, whatever the embedding models and retrieval techniques we used, the hit-rate of the retrieval will drop a lot.

In response to this concern, we commit to: (1) describe the key prompting strategies we used to implement URI to the revised paper; (2) add the above results to the revised paper.

Q2: Generalizability Across Domains.

Thanks for the valuable question. We acknowledge applying URI in just single domains might raise concerns about the generalizability of the methodology. In response to this concern, we build a new proof-of-concept benchmark based on the classic Tie Tac Toe game (TTT) and verify URI. We kindly recommend the review to check the details of the results in the first section of the global response letter.

Q3: Complexity of Implementation

We acknowledge the reviewer's concern on the complexity of the implementation. In response to the concern, we would like to use a high-quality open-source code to keep this study easy to follow and reproduce. We kindly recommend the review to check our open-source plan in the second section of the global response letter.

Q4: Quality of Textual Resources

Q4.1: How do you plan to address potential limitations

The effectiveness of the policy derived by the URI framework fundamentally hinges on the quality of the textual data. It is indeed important that the textual resources should cover sufficiently the dynamics, policy, and rewards of the targeted environment so that relevant knowledge and imaginary data can be extracted to train the policy, otherwise it is impossible to derive a good enough policy. There are several potential approaches to address this limitation:

1. Multimodal Data Integration: Beyond purely textual data, additional modalities such as tutorial voices, demonstration videos, and replays can be incorporated. By employing advanced multimodal large language models, these diverse forms of data can be processed in a manner akin to the current handling within the URI, thereby augmenting the knowledge base and enhancing the robustness of policy learning.
2. Utilization of Real Interaction Data: During the introspection phase, incorporating real interaction data with the target environment, rather than relying solely on simulated data generated by large language models, can enhance policy learning. This mixed-data approach can be used to further fine-tune various modules within the URI framework, potentially boosting overall performance. However, one might develop new techniques to utilize both sources of data for better policy learning.
3. Injection of Prior Knowledge: The URI framework allows for the integration of human expert knowledge at different stages of the pipeline. Experts can provide specific code knowledge representations/formulations/templates when generate code-based knowledge. We can also provide constraints during the rehearsal process to enhance stability and realism.

Q4.2: Criteria for Selecting and Evaluating Resources

Evaluating the quality of training data remains a challenging issue across machine learning community. There are possible methods [1-3] that might guide our evaluation. These studies suggest frameworks and empirical strategies that can be adapted to evaluate the relevance and quality of textual and multimodal data for training purposes. However, given the significant modality gap between textual data and the neural network parameters in PLfB topics—and the largely unsupervised nature of our learning process—this is not a trivial problem. This complexity requires innovative approaches for assessment.

[1] QuRating: Selecting High-Quality Data for Training Language Models

[2] An Empirical Exploration in Quality Filtering of Text Data

[3] DoGE: Domain Reweighting with Generalization Estimation

We believe these discussions are valuable for the research community. In response to this question, we commit to include them in Appendix F of the revised paper. Additionally, we would like to highlight that many real-world decision-making scenarios have an abundance of textual tutorial resources, such as medical diagnosis, financial trading, software development, and educational tutoring. Thus, even in domains with rich textual resources, the topic remains broad and highly valuable.

Q1: If there are more domains where the authors could experiment with different data sources, the readers would have better expectation of the model.

We agree that applying URI to more domains would strengthen readers' understanding, expectation, and belief in it. Thus, we build a new proof-of-concept benchmark based on the classic Tie Tac Toe game (TTT) and verify URI. We kindly recommend the review to check the details of the results in the first section of the global response letter.

Q2: Concern about the complex framework, which has many components. This on one hand is the novelty of this paper, however, it also adds to the difficulty of replicating this method, especially in other domains.

We acknowledge the reviewer's concern about the complexity of the implementation. In response to the concern, we would like to use high-quality open-source code to keep this study easy to follow and reproduce. We kindly recommend the review to refer to the second section of the global response letter for our open-source plan. We will also continue to improve the text to make the whole paper more accessible.

Q3: Can the URI framework be iterated -- collecting data online and summarize new knowledge. And the knowledge can be passed on to the next iteration of URI.

It is an open question but we truly believe the future of URI should be iterative. It is intuitive since even when humans learn a new skill from books, it is not enough to be competent in tasks by just learning from books. We need real-world practices to bridge the knowledge-utilizing gaps in books. However, it is non-trivial to design a practical method to fully utilize the online data to improve the URI pipeline. We have made a brief discussion about such an extension as a future research direction in Appendix F. We are excited to dive deeper into this as future work.