FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making

We sincerely thank the reviewer for their insightful comments and valuable feedback on our paper. We are delighted to receive recognition of our method's effectiveness, robustness, and scalability through extensive experiments, and for acknowledging that our claims are supported by convincing evidence with well-justified evaluation criteria (baselines/metrics). Please find below our detailed responses to the reviewer’s comments.

1. Novelty concerns: We appreciate the reviewer's comparison between FOUNDER and LEXA. While both methods utilize world models and temporal distance prediction, their technical contributions differ fundamentally. LEXA focuses on discovering diverse goals and learning to achieve them during online exploration, enabling adaptation to image goals for downstream tasks. In contrast, the core idea of FOUNDER is to ground foundation model (FM) representations (from multimodal tasks) into the world model (WM)'s latent space through explicit mapping functions, operating entirely offline. This enables physics-consistent translation of VLM embeddings into actionable WM states. It should be noted that world model and temporal distance predictor in FOUNDER only serve as tools to implement this paradigm, whereas the core contribution resides in bridging the gap between FM's general knowledge with WM's physical dynamics, establishing a new methodology for open-ended task solving and setting FOUNDER apart from prior methods using VLMs in RL.

2. Hyperparameter ablation: We appreciate the reviewer's feedback regarding the hyperparameter settings. For frame window size (k=8), this choice strictly aligns with two critical considerations: (1) InternVideo2's pretrained temporal encoder requires fixed 8-frame inputs, as validated in its implementation. (2) Retained GenRL's 8-frame configuration for fair comparison. For the kl weight β, we use β=1.0 by default. To address your concerns, we conducted an ablation study on the Cheetah and Kitchen domains to validate their sensitivity, and the results are on our website. We find that FOUNDER is not sensitive to the kl weight parameter, while 1.0 is the best choice.

3. Goal sampling in behavior learning: In fact, the goal state zg of a given video/text task during behavior learning is sampled from the goal distribution inferred from the video/text VLM embedding, e.g., using the learned mapping function: zg∼Qψ(⋅∣eg). This is discussed in Equation (6) and Appendix C.1.

4. Question about temporal distance learning: We thank the reviewer for raising this valuable point. Learning a temporal distance predictor from quasi-static data may pose risks. However, the world model provides proper regularization to the input states, and such datasets are rare in real applications. To address your concerns, we provide experiments on learning a temporal distance predictor on the Stand datasets, where trajectories are generated by the expert policy in Walker Run, and validate the predictor's accuracy in predicting the same WM states as near-zero temporal distance. The results on our website show robustness of temporal prediction in such settings.

5. Out-of-distribution tasks: In section 5.2, we evaluate the performance of FOUNDER on out-of-distribution video tasks, where the visual appearance of task video prompts differs significantly from the agent's environment observations and training data, including cross-embodiment and cross-viewpoint settings on DMC and Kitchen, and FOUNDER demonstrates superior performance than other baselines. Also, in Sections 5.1 and 5.4, we test FOUNDER on out-of-distribution language tasks like Cheetah Flipping and Hunt Sheep, where the training dataset does not contain buffers of these tasks, and the performance of FOUNDER exhibits its generalization abilities. To further address your concerns, we present experimental results on real-world video tasks on our website using videos provided in GenRL's code repository, and FOUNDER also demonstrates solid performance compared to GenRL.

We sincerely thank the reviewer for their valuable feedback on our paper. We are delighted to receive recognition of our method's sound experimental design and solid experimental results. Please find below our detailed responses to the reviewer’s comments.

The influence of temporal distance: We appreciate the reviewer for raising these important questions regarding temporal distance (TempD). Generally, the role of TempD-based rewards is to enhance temporal awareness and provide task-completion information to the FOUNDER agent.

From Table 1, using cosine similarity between VLM representations of task and observations as rewards, FOUNDER w/o TempD already performs well on static tasks (Stand tasks and Kitchen tasks). However, FOUNDER w/o TempD struggles in dynamic tasks (Walk, Run). We observe that in dynamic tasks, the agent can perform well at the very beginning of the behavior learning stage, but its performance deteriorates as training progresses, eventually reaching very low performance by the end. We also discovered that the real return and pseudo return curves exhibit completely opposite trends during behavior learning. The real performance worsens as the agent maximizes the pseudo reward, causing a reward hacking problem. We then visualize the resulting trajectories of the policy at the early stage and the final policy. Despite performing well at the beginning, we find that at the end, the agent’s behavior becomes more static. The agent may stay at its initial position, appearing as if it is running or walking, but in reality, it is only 'running' or 'walking' at the original place without progressing forward, or moving forward at an extremely slow pace. This explains the poor final performance of FOUNDER w/o TempD in Table 1. These training curves, trajectory visualization, and detailed analysis can be found on our website.

We conclude that reward functions based on cosine similarity or other direct distance metrics may lead the policy to mimic the visual appearance of the goal, while overlooking the underlying task semantics and multi-step movement, particularly in dynamic tasks like running or walking. Since the FOUNDER-based method maps the target sequence to a single goal state in the world model, we hypothesize that using direct distance functions may result in a lack of temporal awareness.

At this time, the role of TempD-based rewards is to provide temporal information and crucial task-completion information to the agent, with a much lower training cost than GenRL's complex sequence matching. In Table 1, FOUNDER then avoids the reward hacking problem and achieves superior performance on dynamic tasks like Walk and Run, compared to GenRL and FOUNDER w/o TempD.

Moreover, Table 1 shows that incorporating TempD will not only benefit temporal dynamic tasks but also match or surpass the performance of FOUNDER w/o TempD on static tasks. Kitchen Burner is the only 1 of the 9 static tasks in Table 1 where FOUNDER is outperformed by FOUNDER w/o TempD, which does not constitute statistical significance.

Furthermore, for Minecraft tasks, FOUNDER w/o TempD performs surprisingly good, and incorporating TempD could weaken the performance on several tasks. We hypothesize that the data quality and the stochasticity nature of the Minecraft environment may have a negative impact on the learning process of the TempD predictor. When temporal predictions become noisy, falling back to cosine similarity serves as a safer baseline, and using visual similarities as pseudo-rewards in short-horizon Minecraft tasks is beneficial and will not cause problems on Walk or Run tasks in DMC. However, when TempD learns accurate temporal dynamics, its temporal credit assignment mechanism provides provable advantages in reward-free multi-task RL, as is proved in prior works [1-4]. We will incorporate approaches discussed in our rebuttal with Reviewer 2yAX, as well as techniques in [1], for better TempD learning under stochastic and complex environments.

[1] Myers V, Zheng C, Dragan A, et al. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making[C]//International Conference on Machine Learning. PMLR, 2024: 37076-37096.

[2] Park S, Kreiman T, Levine S. Foundation Policies with Hilbert Representations[C]//International Conference on Machine Learning. PMLR, 2024: 39737-39761.

[3] Park S, Rybkin O, Levine S. Metra: Scalable unsupervised rl with metric-aware abstraction[J]. arXiv preprint arXiv:2310.08887, 2023.

[4] Mendonca R, Rybkin O, Daniilidis K, et al. Discovering and achieving goals via world models[J]. Advances in Neural Information Processing Systems, 2021, 34: 24379-24391.

We sincerely thank the reviewer for their insightful comments and valuable feedback. Below are our responses to each of their concerns:

Universally applicable claim: Our claim of task-agnosticism refers to FOUNDER’s architecture—world model, mapping function, and reward generator—enabling deployment across downstream tasks specified via text or video inputs without architectural changes. This highlights its broad applicability, not guaranteed universal performance. We appreciate your feedback and will clarify this in the final version.

Open-endedness: In our context, “open-endedness” refers to the agent’s ability to ground and solve user-defined tasks from free-form prompts via multimodal interfaces, without predefined reward functions. This differs from definitions emphasizing self-evolving exploration or unbounded learnability. FOUNDER achieves this capability by jointly learning the WM, mapping function, reward generator, and downstream policy from offline data. This definition aligns with similar works [1-3]. We will clarify this in the final version.

Performance of GenRL: Our reproduction of GenRL follows its official code and datasets. Our reproduced results (GenRL_ours) align with GenRL’s arXiv v1 metrics on in-distribution DMC tasks but underperform on Kitchen tasks, likely due to domain properties. Despite GenRL v2’s improved performance, its implementation was not updated. FOUNDER outperforms GenRL in both cases.

Effectiveness of FOUNDER's components: Our key innovation is grounding VLM representations into WM states without GenRL’s costly sequence matching, enhancing task understanding beyond reward correction. As shown in Table 1, FOUNDER w/o TempD outperforms GenRL on static tasks (e.g., stand, Kitchen), while it requires TempD for dynamic tasks (e.g., Walk, Run) to improve temporal awareness. TempD efficiently provides temporal structure at a much lower computational cost than GenRL’s sequence matching. TempD’s addition to GenRL does not improve its performance, confirming FOUNDER’s architecture drives success.

The “+1 reward” heuristic: The +1 operation smooths rewards, improving learning. Experiments show agents can achieve similar results without this shaping given more training steps. Training curves comparing shaped and unshaped rewards are available on our website.

Minecraft Experiment: High variance in Minecraft experiments is mainly due to the environment’s stochasticity. To better reflect statistical significance, we now report 95% confidence intervals and provide clearer learning curves on our website. FOUNDER matches GenRL on 2/5 tasks and outperforms it on the remaining three.

Comparison with model-free baselines: We directly compare HILP, a multi-task model-free approach, and TD3, the strongest single-task model-free baseline per GenRL. As detailed in our response to reviewer e54P, these comparisons justify the need for a model-based approach, with FOUNDER achieving a strong balance between performance and efficiency.

Limitations related to the temporal distance predictor:

1. The predictor learns with a predefined sequence length during training, but during testing, it outputs the predicted temporal distance independent of sequence length.
2. Temporal distance excels in cross-domain tasks and metric evaluation, where GenRL and cosine similarity methods may fail due to reward hacking. While extreme OOD scenarios may still pose challenges, FOUNDER’s success shows its robustness in capturing deep task semantics rather than relying on brittle visual correlations.
3. We appreciate the reviewer’s insight on mining negative samples. We follow LEXA [4], but acknowledge the risk of mislabeling. This can be mitigated by filtering out high-similarity negative pairs or actively selecting challenging cross-sequence pairs via feature-space searching. We will include an ablation on this in the final version.

Limitations from offline data quality: We agree with the reviewer that dataset properties limit reward-free offline RL methods and have discussed this in Section 6. We welcome further discussion on dataset dependency concerns, as the reviewer did not fully elaborate on the complete questions.

[1] Fan L, et al. Minedojo: Building open-ended embodied agents with internet-scale knowledge. NeurIPS 2022.

[2] Team O E L, et al. Open-ended learning leads to generally capable agents. arXiv preprint.

[3] Qin Y, et al. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. CVPR 2024.

[4] Mendonca R, et al. Discovering and achieving goals via world models. NeurIPS 2021.

We sincerely thank the reviewer for their insightful comments and valuable feedback on our paper. We are delighted to receive recognition of our method's solid performance and strong task generalization demonstrated by extensive and detailed experiments. Please find below our detailed responses to the reviewer’s comments.

Other model-free baselines: Prior work (e.g., GenRL) has proven that traditional model-free offline RL methods (IQL, TD3+BC, TD3) perform poorly in this setting compared to model-based approaches. These methods rely on zero-shot pseudo-rewards generated by Foundation Models (FM) through cosine similarity between FM representations of task prompts and observations. However, FM rewards are not explicitly grounded in the embodied domain, limiting their effectiveness. Additionally, traditional model-free methods require training from scratch for each task, making them inefficient and impractical for multi-task solving.

However, to ensure a comprehensive evaluation and address your concerns, we now include HILP [1], a multi-task model-free method that achieves strong zero-shot RL performance in goal-conditioned RL, and TD3, the best-performing single-task model-free baseline according to GenRL’s experiment. We assess their performance across eight tasks in the Cheetah and Kitchen domains, alongside GenRL for a clearer comparison. The results are on our website. We find that FOUNDER consistently outperforms both single-task and multi-task model-free baselines, and our prior claim is reinforced. We sincerely appreciate the reviewer’s suggestion regarding model-free baseline comparisons.

Efficiency and computational costs: We appreciate the reviewer's emphasis on computational efficiency and provide a comparison of the four typical approaches mentioned. All experiments were conducted on an RTX 3090 GPU. For pretraining, training the MFWM in GenRL for 500k gradient steps requires approximately five days, whereas FOUNDER reduces this overhead to about three days by avoiding the seq2seq-style generative sequence modeling used in GenRL. Pretraining the Hilbert representation and foundation policy in HILP for 500k gradient steps takes around 30 hours.

For downstream tasks, training the actor-critic on a given task prompt for 50k gradient steps takes under five hours for both GenRL and FOUNDER. In the case of model-free methods, HILP enables zero-shot adaptation to downstream tasks, whereas single-task model-free methods like TD3 require approximately seven hours to train from scratch for 500k gradient steps. Overall, we find that FOUNDER strikes a strong balance between performance and learning efficiency.

Scalability of FOUNDER We have included a brief discussion on the scalability of FOUNDER to long-horizon tasks in Section 6. While our current experiments focus on short-horizon tasks, FOUNDER’s architecture is inherently designed to support long-horizon reasoning through two key mechanisms: (1) the world model’s temporal abstraction via latent state transitions and (2) the foundation model’s semantic grounding, which enables hierarchical task decomposition. To fully unlock this potential, there may exist three enhancements: (i) integrating more powerful sequence modeling architectures (e.g., [2]) or continual learning methods (e.g., [3]) a for persistent context tracking, (ii) implementing FM-guided curriculum learning [3] to phase in complex tasks, and (iii) developing a meta-controller for dynamic subgoal generation. We thank the reviewer for raising this concern, and we will explore it in our future work.

[1] Park S, Kreiman T, Levine S. Foundation Policies with Hilbert Representations[C]//International Conference on Machine Learning. PMLR, 2024: 39737-39761.
[2] Samsami M R, Zholus A, Rajendran J, et al. Mastering memory tasks with world models[J]. arXiv preprint arXiv:2403.04253, 2024.
[3] Feng T, Wang X, Zhou Z, et al. EvoAgent: Agent Autonomous Evolution with Continual World Model for Long-Horizon Tasks[J]. arXiv preprint arXiv:2502.05907, 2025.