Thank you for your feedback on our work. Below, we would like to address your questions and concerns.

P1 Additional baselines.

We understand that a key concern in your review is the lack of additional baseline comparisons. However, to the best of our knowledge, no existing standard RL methods operate under the exact same setting as TEDUO beyond those already included in our benchmarking. Prior work typically requires either access to an online environment, something we explicitly rule out due to our focus on offline training paradigms or relies on labeled demonstration data, whereas we work with only an unlabeled set of state-action transitions.

• New result: As suggested by reviewer AzTQ, we include an additional comparison with DeepSeek-R1, not available at the time of our original submission. We have incorporated its performance into our benchmark; results under this link.

If the reviewer is aware of any other works that can be applied in our setting, we would be eager to include them in our comparisons to further strengthen our evaluation. We would be grateful for any suggestions!

P2 The complexity of tasks and environments.

We appreciate the reviewer’s concern regarding the complexity of our evaluation environments. BabyAI and WebShop were deliberately chosen as two distinct yet illustrative environments: a grid-based world and a text-based web interaction task. This illustrates TEDUO’s adaptability across symbolic environments of different kinds. Additionally, as detailed in Appendix C.1, WebShop presents significant challenges, such as dynamic UI-based actions and linguistic action requirements, making it a non-trivial testbed.

TEDUO's modularity makes it readily generalizable to other symbolic environments, as outlined in Appendix C.2 (Guide for Practitioners). While environments such as RLBench habitat and Minecraft are indeed complex, they are also image-based, which diverges from TEDUO’s focus on symbolic, text-based tasks. As acknowledged in our limitations section, continuous control tasks like RLBench may not be best suited for LLM-based agents. However, we recognize the potential for extending TEDUO’s applicability via, e.g., VLMs. We leave such extensions for future work.

While scaling to more complex settings is an important goal, it is a common practice in RL research to focus on simple environments for a detailed and interpretable analysis. Notably, many works recently published at top-tier venues still focus on toy environments. For instance, this includes the relevant related work [1] cited by reviewer azTQ, which uses a card game as their primary environment for experimentation, and the related works cited by reviewer vbjR [1,3,4], which use environments much simpler than Webshop. Our focus on these settings allowed us to better understand TEDUO’s core mechanisms, providing a foundation for future extensions to more complex settings.

Finally, while BabyAI and WebShop may appear toyish, we respectfully argue that the significance of our results goes beyond the complexity of these benchmarks. To the best of our knowledge, TEDUO is the first method capable of learning generalizable, language-conditioned policies from fully unlabeled data, which we find a significant and exciting result. The presented results demonstrate the promise of combining LLMs with RL-based approaches to enable flexible generalization.

• New Results: To deepen the argument about flexible generalization, we have performed a new experiment demonstrating TEDUO’s generalization to larger, unseen BabyAI grids after training on smaller grids. Performance remains robust for grids three times larger than the grids seen during training. Results are available under this link : link.

P4 How are the goals in $\mathcal{G}^{tr}$ obtained?

The goals are manually designed (see Appendix C.8.1 and C.8.2, paragraphs “Goals” for details).

P5 What is the temperature setting for the LLM?

We used the default temperature of the Llama-8B-Instruct model: 0.7 (both for TEDUO-fine-tuned and baselines).

Motivated by your suggestion, we ran an additional small-scale experiment using online evaluation. Results, available at this link, show a slight improvement for temperature=1.2. We interpret this as a higher temperature promoting more diverse actions, helping the agent not to get stuck in suboptimal behaviors.

Please note, tuning of the temperature is feasible when simulators are available. In offline RL scenarios, model selection is complicated, and hyperparameter tuning may not be possible.

We sincerely appreciate the reviewer’s time and thoughtful feedback. We hope that our clarifications and new results address your concerns and strengthen the contributions of our work. Please let us know if there are any additional points that would benefit from further elaboration!

We thank the reviewer for their thoughtful feedback and valuable suggestions. Below, we address each of the key concerns and clarify aspects of our work.

P1 Hindsight Labelling

We acknowledge that LLM-based hindsight labeling itself is not novel, and we did cite prior work (Appendix A.2). The reviewer’s suggested references (MineDojo, LMA3) will also be added. However, these operate online, while TEDUO is fully offline. Our contribution also lies in scaling LLM-based reward functions by approximating them with NNs to reduce LLM calls. Action: We will refine our contribution statement to avoid overclaiming and emphasize these specific contributions.

P2 LLM as a Goal Labeler

We clarify that the results presented in B.5 report the accuracy of the rewards predicted by the NN reward approximators with respect to the environment ground-truth rewards. Action: We have additionally included the accuracy, precision, and recall metrics comparing LLM reward labeling to ground truth rewards. See this link for results.

P3 Appendix B.2

We acknowledge the reviewer's difficulty in interpreting Appdx. B.2, and we agree that the definition of $|\mathcal{S}_0^g|$ was unclear. Action: We have provided a more detailed definition and the motivation behind this metric. The revisions can be found under this link: link. Thank you for drawing our attention to this part!

P4 Experimental designs

Unlabeled Demonstrations The claim that "none of the experiments leverage a true dataset of unlabeled demonstrations" is incorrect. We always act as if the data collection policy were completely unknown. Even if the policy used is a mixture of goal-conditioned policies, we do not assume knowledge of which trajectory has been generated with respect to which goal.

Semantically Distinct Goals We define two "semantically distinct" instructions as two goals with different goal states, rather than paraphrased versions of the same task. (E.g., ”Pick up a red box” is distinct from “Pick up a yellow key” and is semantically equivalent to “Collect a red container”). While prior works have mostly focused on paraphrased instructions, we specifically test whether TEDUO can generalize to goals that differ in key task attributes. While this may seem incremental, we note that conventional RL approaches have struggled with this form of generalization. Please also consult P2 in the answer to reviewer zuF6 for new results on generalization to more complex tasks.

Construction of $\mathcal{D}^{SFT}$. We confirm that no additional interaction with the environment occurs in the construction of $\mathcal{D}^{SFT}$; an empirical transition function is used. This is due to our focus on purely offline RL.

P5 Fine-tuning with Imitation Learning.

See P5 in the answer to reviewer vbJR.

P6: Directly applying Offline RL

We appreciate the reviewer’s suggestion of applying offline RL directly to the LLM. While promising, this poses challenges due to scale and the natural language goal-conditioned setting.

ILQL is an interesting adaptation of Implicit Q-Learning, but it targets standard RL tasks with a single reward function, not GCRL, which requires learning multiple reward functions and generalizing across tasks. Scaling value estimation in GCRL remains an open challenge.

Recent efforts to adapt IQL to GCRL (excluding the natural language aspect) have succeeded only in maze navigation [1]. To our knowledge, ILQL has not been extended to natural language GCRL, likely due to scalability issues. We agree this is a valuable research direction.

P7 The necessity of Offline RL

We would like to highlight that filtered BC significantly improves performance (from an average score of 5 to 50 out of 100), demonstrating its effectiveness. A detailed discussion on the necessity of offline RL can be found in P5 in the answer to reviewer vbJR.

P8 Performance of offline RL Policies vs. Final Results

We would like to clarify that the average performance of the offline RL policies is low due to evaluation on unseen initial states, i.e. states that have not been visited in $\mathcal{D}$. In such states, the Q-learning policies can only take random actions, leading to poor performance. For fine-tuning the LLM agent, we only use successful transitions to perform imitation learning via SFT. That is, $\mathcal{D}^{SFT}$ only includes trajectories that lead to goal achievement, which we can generate without access to online interaction with the environment by relying on the empirical state transition function and the learned reward functions.

P9 State abstraction

See P5 in the answer to reviewer aztQ.

We thank the reviewer for their detailed and insightful feedback. We are confident that the revisions we plan to incorporate will further clarify and strengthen the paper and we are happy to answer any further questions you may have!

Thank you for your enthusiastic review and constructive feedback! We’re delighted by your positive assessment and have addressed your questions below to strengthen the manuscript further.

P1 Task Complexity

See P2 in answer to reviewer zuF6, showing new results on generalization from simple to more complex tasks.

P2 Questions: Base vs. instruction-tuned models

1. We used Llama-3-8B-Instruct throughout all experiments (Appendix A.1), which we will clarify in the main text to avoid confusion.
2. Since we already use the instruction-tuned version, this question does not apply.
3. Similarly, since our experiments already use the instruct model, we think a comparison with the base model is not necessary.

P3 Invalid action rate

Pure RL policies (e.g., GCRL, tabular Q-learning policies) are designed to only take actions that have been seen in the training dataset (unless they are in a previously unseen state where the action is taken at random), minimizing invalidity. In contrast, in TEDUO, the LLM’s prior over the action space (which may contain invalid actions) is merged with the learned policies via SFT. While, on the one hand, this enables the LLM to flexibly generalize to new tasks and environments, it may occasionally lead to proposing invalid actions. As demonstrated in our experiments, the invalid action rate increases as the inference setting diverges from the training distribution; a non-fine-tuned LLM has a very high rate of invalid actions. This reflects a trade-off between generalization and memorization of the observed behaviors. When the agent is in a previously seen state, a potential workaround could integrate action masking by pruning options deemed invalid by the Q-learning policies. Action: We will discuss these observations in the limitations section of the revised manuscript, particularly in relation to high-stakes environments where safety is a concern.

P4 Paper Title

We appreciate the reviewer’s feedback regarding our title. While it was not LLM-generated (we promise!), we are open to suggestions for improving clarity in the camera-ready version.

P5 Training directly on collected trajectories vs. TEDUO.

We interpret the reviewer’s question as suggesting an alternative approach where the LLM is fine-tuned directly on observed trajectories. However, this approach has several limitations:

Firstly, the original data collection policy used to generate the set of trajectories used for subsequent offline RL may be highly suboptimal with respect to the test time goals. In fact, it can even be random (see Appendix B.2. for an experiment with randomly collected observational data).

Secondly, the generated trajectories a priori are not goal-labeled. While we could, in theory, fine fine-tune the LLM on sequences: $s_0, a_0, s_1, a_1, \ldots$, the LLM could not be made aware of which goal is the given sequence solving. This challenge is addressed by the hindsight labeling step of our pipeline.

Finally, skipping the policy learning stage and fine-tuning the LLM on just the trajectories matched with the goals after step 1 is expected to perform strictly worse than fine-tuning the LLM on the Q-learning policies. This is a consequence of the suboptimal nature of the data collection policies. Importantly, this Q-learning step is also the most computationally inexpensive part of our pipeline, as it does not involve any LLM calls, so including it only marginally increases the overall compute.

P6 Concurrent work

Regarding the concurrent work cited by the reviewer [1], we find it an interesting idea, but we also highlight some key differences:

• In the first part of this method, the collection policy (an LLM) is assumed to be good enough to generate a large number of trajectories with positive reward. (in our paper the step 2 (offline RL) goal is to obtain more positive trajectories based on the suboptimal offline data).
• In the second part, they shift to online interaction, generating trajectories from their trained models, whereas TEDUO remains strictly offline throughout.
• This method does not handle unlabelled observational data, while TEDUO explicitly addresses this by leveraging hindsight labeling to infer goals. As a result, the approach in [1] is not applicable to the data regimes considered in our work.

To teach the LLM [1] uses DPO (with both positive and negative trajectories in a contrastive way) while we only teach the positive trajectories via SFT. This alternative approach is interesting and could be used in TEDUO step 3. We are happy to cite this work in the related section of the updated version of our manuscript.

Thank you once again for your feedback. We’re thrilled by your excitement about TEDUO’s potential and hope these revisions solidify its contribution. Thank you for your support!

Thank you for your review and constructive feedback. We appreciate your engagement with our work and have carefully addressed your concerns below:

P1 Task Complexity and Scalability

See P2 in the answer to reviewer zuF6, including new results on generalization from simple to complex tasks.

P2 Scalability of Q-Learning

Tabular Q-learning was used in our main experiments as it suits the BabyAI environment. However, TEDUO’s pipeline is agnostic to the choice of the learning algorithm in step 2. For larger state spaces, it can incorporate more scalable offline RL methods like DQNs, which our ablation study (Appendix B.4) shows perform comparably on BabyAI. In WebShop, we instead used filtered behavioral cloning, as Q-learning would be ineffective due to the limited number of trajectories in comparison to the dimensionality of the state space. WebShop was intentionally chosen to demonstrate TEDUO’s flexibility, reinforcing that it is agnostic to the policy-learning algorithm.

P3 Trajectory Rewriting and Offline RL

While trajectory rewriting indeed cannot introduce new exploration, development of flexible offline RL methods is critical for real-world settings where exploration is unsafe or impractical (e.g., robotics, healthcare). TEDUO’s focus on offline training aligns with such requirements. That said, we find the extensions of similar hybrid approaches employing LLMs as RL agents in online settings a highly promising area of research, which is beyond the scope of this work.

P4 Additional RL baselines and comparison to DeepSeek R1

We agree that comparisons to state-of-the-art baselines are essential. However, as TEDUO focuses on offline language-conditioned RL, to the best of our knowledge, there are no directly comparable prior works beyond the ones already included in our benchmarking. We are open to suggestions regarding additional baselines suited for the offline language-conditioned RL setting. Regarding generalization to more complex tasks, we have now extended our evaluation with a new experiment demonstrating our method’s success (see P1). Please note, testing RL agents on more complex tasks than the ones seen at training time is non-conventional and we find the presented result very exciting.

Comparison to DeepSeek R1. The reviewer rightly notes the success of recent works employing GRPO for online policy improvement in the context of LLM training. However, such methods require access to real-time experimentation to enable policy rollouts. The focus of this paper is on learning from passively collected observational datasets. We find employing ideas similar to the ones observed in Deepseek R1 a very promising direction for follow-up works.

• New result: We included a comparison of TEDUO to the baseline of prompting Deepseek R1, see P1 in the answer to reviewer zuF6.

P5 Question: State abstraction

We agree with the reviewer’s suggestion to clarify the role and implementation of state abstractions. State abstraction serves two purposes: (1) transforming the states from their initial modality into text and (2) filtering all irrelevant information to achieve the goal. For a web agent like Webshop or WebArena, the first purpose is achieved by transforming the HTML code of the webpage into a curated text report, where possible actions (button/search bar, etc.) are identified. The second purpose is achieved by removing irrelevant information such as advertisements or clearly irrelevant buttons etc. In our case for Webshop, the first part is natively supported by the environment, and the second part is not needed due to the noiseless nature of the provided state.

P6 Question: Direct (online) RL training

TEDUO prioritizes offline RL because many real-world applications (e.g., healthcare, education) prohibit online exploration. While online RL excels in simulated settings, our framework addresses practical constraints. The cited work ([1]) focuses on online RL, which is orthogonal to our setting.

[1] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

Thank you for suggesting the areas for improvement, which has helped us refine and better articulate TEDUO’s contributions. Your points inspired us to:

• Push generalization further with new results showing the promise in TEDUO’s ability to train LLM agents to generalize to new, more complex tasks.
• Clarify scalability by emphasizing TEDUO’s compatibility with more scalable offline RL algorithms, like DQNs.
• Highlight the focus on real-world data-constrained applications and discuss exciting areas for future work, extending LLM+RL hybrids to online settings.

We hope you’ll find our revisions compelling and reconsider your score. Thank you for helping us in making a meaningful contribution to the field.