SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models

We thank the reviewer for the constructive review. We appreciate that you found our paper comprehensive, the introduced concepts interesting, and our experimental design sound.

VLM-based exploration baselines

Existing VLM-based exploration methods (Omni, Motif, ELLM) rely on assumptions that limit their applicability to our setting, such as requiring expert demonstrations (Motif), language-grounded environments (Omni, Motif, ELLM), or high-level actions (Omni, ELLM). In contrast, SENSEI operates in a more general RL setup without these constraints.

Motif also uses an initial offline expert dataset. Omni requires access to the full task list and corresponding rewards. Motif and ELLM both rely on event-counting to bias novelty. In contrast, we show how novelty and semantic interestingness can be seamlessly combined with minimal assumptions, enabling deployment in low-level control environments while preserving scalability.

Additionally, none of these methods use model-based RL, which is central to SENSEI’s sample-efficient exploration. For fairness, we adapt Motif into our VLM-Motif baseline. Our results show SENSEI outperforms VLM-Motif, highlighting the benefits of dynamic reweighting and the importance of incorporating information gain within a model-based approach. We will add this discussion to the final version.

Offline VLM usage

For details on why offline VLM usage is necessary/preferred and on our distillation procedure, please see our answer to Reviewer 7mHN (Section: Approximating human interestingness).

Diversity of Envs and corresponding interestingness

There is also a question of how large and diverse these environments are, whereby the goal state may be the only "interesting" state there is.

We now include a diverse environment: Pokémon Red. It requires navigating a large map and mastering battle mechanics to catch/fight wild Pokémon and defeat Trainers. Here, interestingness is multi-faceted (map progression, Pokémon collection, etc.), and SENSEI shows consistent gains. Please see our response to Reviewer 7mHN and our rebuttal webpage: https://sites.google.com/view/sensei-icml-rebuttal.

Pre-Collected Dataset

SENSEI uses pre-collected data, unlike Plan2Explore, but our dataset size is chosen to ensure coverage of interesting states—not to be minimal. Crucially, SENSEI remains effective with significantly less data. In a new ablation, we show that in MiniHack-KeyRoom, using just ¼ of the dataset (25k samples) yields similar interaction statistics (Webpage Figure E5).

In real-world settings, various offline robotics datasets (e.g., OpenX-Embodiment, TriFinger RL) contain expert and non-expert control data, making SENSEI directly applicable. Alternatively, SENSEI could begin with exploration purely via $r^{dis}$, then distill the reward from collected data. We appreciate this suggestion and will include the discussion in the final version. It aligns well with our generational SENSEI perspective, further supported by the Pokémon experiment (see response to Reviewer 9xtC).

Alignment Between Goals and "Interestingness"

If a task’s goals or subgoals don’t align with "interestingness", SENSEI relies on epistemic uncertainty ($r^{dis}$) to discover them. In practice, this has not been limiting.

For instance, in MiniHack-KeyRoom, the goal is to reach a staircase (Fig. 4, top, img 5), which GPT-4 does not consider especially interesting. Standing near the exit yields lower semantic rewards (point 5) than standing near the key and door (points 3&4). Still, SENSEI reliably solves the task and collects more rewards than Plan2Explore by reaching the staircase.

Interestingness vs Information content

Interestingness captures semantic relationships between entities, which may not be reflected in simple information content heuristics.

For example, in MiniHack-KeyChest, compare: (1) “Agent is next to a locked chest with a key in the inventory” and (2) “Agent is next to a locked chest with no key”. Both have similar information content, but (1) is more interesting due to the key-chest relation.

Can interestingness be adjusted based on existing information (e.g., a new object is more interesting than a familiar one)?

This is precisely what SENSEI achieves by combining interestingness with disagreement (information gain). We aim to explore regions that are both interesting and novel.

What kinds of states are "interesting"? How is this distinct from human-like subgoals in earlier work?

We assume VLMs incorporate priors aligned with human preferences. Thus, human-like subgoals would likely be favored by VLMs over random states. However, unlike prior goal-based work, we don’t assume access to a subgoal set. Instead, we use Motif to compute a continuous reward, approximating interestingness.

We hope this response addresses the reviewer’s concerns and we would be happy to clarify further if needed.

We thank the reviewer for their feedback and greatly appreciate that they found our paper well-organized, our experiments thorough, and our evidence clear and convincing. We aim to address the remaining concerns below.

The paper lacks comparisons to recent exploration methods beyond P2X like Curiosity-Driven Exploration via Disagreement or Skew-Fit

First, we note that we compare not only against P2X, but also RND and VLM-Motif—a reimplementation of Motif adapted to our world model and VLM setting. Thus, we compare against 3 strong exploration baselines.

Second, Plan2Explore represents the state-of-the-art in curiosity-driven exploration via disagreement for pixel-based inputs. We are not aware of other disagreement-based methods suitable for comparison.

Skew-Fit, on the other hand, falls into the goal-based exploration paradigm, where goals are sampled to maximize state coverage. Our RND baseline also targets state-space coverage. We believe goal-based methods are orthogonal to our setting. In addition, we note that Hu el al. 2023 [1] shows (Fig. 4 of main paper) that an improved model-based adaptation of Skew-Fit is performing worse or similar to Plan2Explore on a range of environments, making Plan2Explore the stronger baseline to compare to.

Testing SENSEI on a broader range of tasks (e.g., navigation, social interaction) would better demonstrate its versatility.

We have now added a new environment—Pokémon Red—which involves navigation over a large map and interactions with Pokémon and gym leaders. See our response to Reviewer 7mHN for details and our rebuttal webpage (https://sites.google.com/view/sensei-icml-rebuttal) for results.

This paper lacks a detailed methodological comparison with Plan2Explore.

Our method builds on Plan2Explore by adding semantic rewards and dynamic reward scaling. We apologize if this was unclear. We have clarified this connection in the method section and discussion. We also empirically analyze the contributions of both additions via comparisons to VLM-Motif and through ablations on dynamic reward scaling (Suppl. D.6).

In Figure 8, why do you choose observations from view (b) apart from the reasons you mentioned in the paper?

In Robodesk’s default view, the robot arm often occludes objects, presenting challenges for learning. Early tests also showed better VLM annotations from the right camera view, which we therefore used.

However, the definition of interestingness is a very subjective concept. Is the exploration of all tasks considered interesting?

We agree that interestingness is subjective. However, we argue that across cultures, commonalities exist in what humans find interesting—reflected in the large-scale training data of VLMs. The tasks in our environments were all designed by humans, implying inherent human interest.

For annotation with SENSEI-General in Robodesk, we first ask the VLM for a description of the environment and what could be interesting. We observe strong alignment between the VLM’s responses and the environment’s task distribution (Supp. C.3.2).

In Figure 7, SENSEI has already trained for 500K steps before fine-tuning on downstream tasks. Would it be fair to let DreamerV3 train for an additional 500K steps?

Excellent point—our method does spend more time in the environment. However, note that SENSEI does not optimize for any particular task during exploration. Even with additional training, Dreamer, in our experiments, does not match SENSEI’s performance.

For example, in MiniHack-KeyRoomS15, SENSEI solves the task reliably after 300K steps of task-based training (on top of 500K steps exploration), totaling 800K steps. Dreamer, even after 1M steps, does not reliably solve the task across seeds.

To emphasize this further, we now compare SENSEI in a Robodesk task (Upright-Block-Off-Table) after 1M exploration steps to Dreamer trained from scratch. SENSEI solves the task across all seeds in 600K steps; Dreamer only succeeds in one seed after 2M steps. Even counting SENSEI’s 1M exploration, it outperforms Dreamer with 1.6M total steps (see our rebuttal webpage, Figure E4).

Task scenarios like Noisy TV may lead to excessive exploration of novel areas, resulting in exploration failure. Have you considered these?

Indeed, this is a known issue in intrinsically motivated RL. However, due to the ensemble disagreement formulation—used in Plan2Explore and our method—this is not a concern. In stochastic environments, the ensemble predictions converge to the process mean with enough samples, resulting in 0 disagreement and no further reward. We refer the reviewer to [2] for more details.

[1] Hu, E. et al., Planning goals for exploration, ICLR 2023.

[2] Pathak, D. et al., Self-supervised exploration via disagreement, ICML 2019.

We sincerely appreciate your thoughtful feedback and recognition of SENSEI’s novelty, thorough evaluations, and strong empirical evidence. Below, we address your key questions and minor comments.

SENSEI in newly unlocked areas

What happens if exploring a behavior in one area unlocks uncertain behaviors in a previously explored area?

As the reviewer correctly noted, we differentiate two cases: 1) the newly unlocked behavior exists in the initial self-exploration dataset, 2) the behavior is entirely novel.

Case 1 is naturally accommodated by SENSEI, as these semantic relationships are reflected in the learned VLM-Motif. For example, in MiniHack-KeyChest, an agent might initially discover a locked chest before finding the key. Upon acquiring the key, the previously explored chest gains new semantic significance, yielding higher semantic rewards. SENSEI adapts seamlessly, as semantic meaning evolves with discoveries. In contrast, Plan2Explore and other uncertainty-driven methods lack an inherent mechanism for handling such scenarios.

Case 2 connects to our new experiments and the reviewer's question 3:

Second Generation of SENSEI Annotations

Since the distilled reward function is fixed during world model training, what if the agent discovers behaviors or states absent from the initial self-supervised data?

Great point. If an agent encounters entirely novel states, the semantic reward may contain mostly noise, as it’s out of distribution for the trained VLM-Motif. Here, exploration relies on epistemic uncertainty rewards ($r^{dis}$), similar to Plan2Explore. In such cases, we can refine our notion of interestingness with new data. We adopt a generational perspective: run SENSEI for a first generation, then perform a second round of VLM annotations to distill new semantic rewards based on newly explored regions.

We tested this in Pokémon Red, comparing SENSEI-General to Plan2Explore (see Reviewer 7mHN response for environment details and results). The initial SENSEI dataset contains 100K pairs from a Plan2Explore run (500K steps), reaching at most Viridian Forest. SENSEI reaches the frontier of Viridian Forest more often, occasionally breaking into Pewter City, where the first gym is located. However, the first-generation VLM-Motif has never seen this gym and thus relies on information gain to explore beyond Viridian Forest. After 750K steps, we sample 50K new pairs from a SENSEI run and re-annotate them, forming a refined second-generation semantic reward function ($R_\psi$) now informed by previously unseen maps.

Qualitative results (Figure E2) on our rebuttal webpage https://sites.google.com/view/sensei-icml-rebuttal show that second-generation annotations correctly emphasize Pokémon Gyms—unlike the initial generation. This is essential for the agent to further explore the gym and beat the gym leader.

We hope this motivates extending SENSEI to complex domains where new behaviors are unlocked through generations.

Semantic Rewards and Evolving Experience

What happens if what’s considered interesting changes over time?

While our semantic reward $r^{sem}$ is independent of agent experience, we assume a VLM inherently ranks more complex tasks (e.g., stacking) higher than simpler ones (e.g., pushing). However, SENSEI also incorporates epistemic uncertainty ($r^{dis}$) into its exploration reward ($r^{expl}$, Eq. 7), so the agent engages in interesting and yet novel behaviors. As exploration progresses, $r^{dis}$ naturally decreases as the world model improves (Eq. 6).

Consider a case where the VLM assigns equal interestingness to pushing and stacking. Initially, both are uncertain, yielding similar $r^{expl}$. As the agent masters pushing, $r^{dis}$ diminishes, lowering $r^{expl}$ for that skill. The agent then prioritizes stacking—aligning exploration with increasingly complex tasks.

Minor Comments

Human evaluations in Robodesk

In the Robodesk evaluation, it would be useful to include a human evaluation of behaviors to support the claim that “our semantic exploration reward leads to more meaningful behavior than purely epistemic uncertainty-based exploration.”

To avoid overstating, we revise the sentence to: “On average, our semantic reward leads to more object interactions than purely epistemic uncertainty-based exploration.”

Why is SENSEI-General not shown in MiniHack?

The MiniHack prompt is already highly general. We describe it as a “rogue-like game” with a generic description and note the egocentric view, without explicitly mentioning doors, chests. The key is only mentioned if it is part of the agent's inventory (Supp C3.3). This differs from Robodesk, where we specify relevant objects. Given this broad framing, we don’t expect SENSEI-General to yield significantly different results.

We appreciate the reviewer’s insightful questions and welcome any further discussion.

Thank you for reviewing our paper and highlighting that our research direction is important and acknowledging that our experimental results are interesting.

New Environment: Pokémon Red

As per your suggestion, we now apply SENSEI to another environment: the classic Game Boy game Pokémon Red. This is a challenging environment due to its (1) vast, interconnected world and (2) complex battle system requiring semantic knowledge (e.g. Water beats Rock). Strong exploration is essential for progressing toward the main goal: becoming the Pokémon Champion by defeating Gym Leaders. We use the PokeGym v0.1.1 setup [1], based on [2], with:

• Observations: Only the raw 64×64 game screen, unlike previous RL applications which additionally used internal game states or memory [2,3].
• Actions: A 6-button Game Boy action space (Left, Right, Up, Down, A, B).
• Rewards: Event-based rewards following [1].
• Episode Length: Episode length is increased with environment steps: 1k until 250k steps, 2k during 250–500k, 4k during 500–750k, etc. We take actions every 1.5seconds of gameplay.
• Game Start: We skip the tutorial, until the agent can move freely and catch Pokémon, starting after receiving the Pokédex. We use the same checkpoint as [2], beginning with a level 6 Squirtle.

We compare SENSEI-General to Plan2Explore. Both receive extrinsic task-based rewards alongside self-generated exploration rewards. SENSEI's initial dataset is collected using 500k steps of Plan2Explore (100K pairs). We use a general prompt:

'Your task is to help me play the Game Boy game Pokémon Red. I just obtained my starter Pokémon, Squirtle. My goal is to find and defeat the Gym Leader, Brock. What do I need to do, and which areas do I need to traverse to achieve this goal? Keep it very short.'

We generate five different responses from GPT-4 and sample from them uniformly as context for image-based comparisons.

The results are shown on our rebuttal webpage: https://sites.google.com/view/sensei-icml-rebuttal. We plot the histogram of the maximum map reached by the agent at each episode throughout the training run for SENSEI and Plan2Explore (5 seeds). Our results show that SENSEI progresses further into the game, and most notably manages to reach the first Gym in Pewter City, unlike Plan2Explore. Both SENSEI and Plan2Explore explore catching and training Pokémon, with SENSEI typically assembling a party of slightly higher levelled Pokémon than Plan2Explore. In order to succeed in the game you need to have a diverse team with high levels. This means achieving a high party level (higher counts on the right side of the histogram).As Pokémon unlocks new maps over time, we also explore using generations of SENSEI: refining the semantic reward using data from an earlier SENSEI run. See our response to Reviewer 9xtC for details. Thank you for your great suggestion of adding another environment. Let us know if you have further questions about our setup or results.

Approximating human interestingness

We assume “proxy-of-a-proxy” refers to our two-step distillation process of 1) first training motif from VLM annotations and 2) learning to predict semantic rewards in the world model. We understand the reviewer’s concern, but want to emphasize why both design choices are essential. We will explain this here in detail and add more emphasis on these aspects in our updated paper.

1. Distilling a semantic reward function $R_\psi$ from VLM annotations is necessary as it allows us to not deal with a large and slow VLM inside the fast RL loop. Additionally, directly querying VLMs can lead to noisy outputs, which can further derail the agent’s learning.

2. Learning a mapping from world model states to semantic rewards is a necessary design choice when working with latent state world models as we point out in the discussion and detail in Suppl. A. 3.

World models such as the RSSM typically encode and predict dynamics fully in a self-learned latent state. Thus, for a world model to predict $r^{sem}_t$ at any point in time, we need a mapping from latent states to semantic rewards.

This begs the question whether our mapping is the best way to do it. Another option would be to decode the latent state to images and use those as inputs for Motif. However, we believe this has several disadvantages: (1) Inefficiency—image decoding from latent states is expensive and slows down training. (2) Artifacts—decoded images can be blurry or unrealistic, causing a distribution shift and making Motif, trained on real environment images, encounter out-of-distribution errors. We hope this motivates our two-step distillation process. We thank the reviewer for allowing us to clarify this and update the paper to better emphasize this in the method section and discussion.

[1] Suarez (2024): https://pypi.org/project/pokegym/0.1.1/

[2] Whidden (2023), https://github.com/PWhiddy/PokemonRedExperiments,

[3] Pleines et al. (2025): Pokemon Red via Reinforcement Learning