Thank you for your thorough read, your great comments and your helpful suggestions.

Domain specific prompting

Thank you for raising this point and the fantastic suggestion that the “VLM could be tasked with inferring the context from the visual observations”. We have now added a new experiment where we perform annotations in Robodesk without injecting any environment specific knowledge. We instead follow a multi-turn prompting strategy, similar to your suggestion, where we first prompt the VLM with an image of the environment, asking the VLM to describe it in detail. We then ask for it to rank two images based on its own description of the environment (prompt details in Supp. C.3.2). Finally, we empirically show that the semantic rewards of the general prompt version of VLM-MOTIF with zero external knowledge injection behaves qualitatively very similar to its environment specialized prompt counterpart and matches the peaks in “interesting” moments of interactions (Supp. D.4, Fig. 14).

Reliance on initial dataset

It is true that SENSEI reinforces trends in the initial dataset used for VLM annotation, which we analyze for Robodesk in Suppl. D.2. We agree with your analysis of multiple rounds of SENSEI annotations as a remedy, as we had also discussed in our future work section. We believe this is an exciting research direction, and we are happy to hear that you share our enthusiasm.

Computational cost of experiments

It is correct that, for the case you described, the VLM is prompted roughly 140k times. This is indeed computationally costly. However, these computations are completely offline, and do not need to happen at runtime of SENSEI such that they can be parallelized. Furthermore, because the VLM does not need to be prompted while training the world model, SENSEI is relatively efficient during runtime and runs on just one GPU. We add a new section detailing the computational cost of our experiments in Suppl. D.7.

Comments and suggestions

Thank you for the careful reading and catching errors.

I would suggest omitting the parts from your section titles in Section 2, such as "UNLEASH YOUR SENSEI"

We changed this title as it is indeed a bit unprofessional. But we kept some other section titles that we felt could help structure the flow of this section and are not too unprofessional.

However, what is not completely clear to me is how or when it then switches back to continue finding new interesting states?

The switching mechanism is implemented by keeping statistics of the predicted interestingness of states. A quantile of these statistics, $Q_k$ for the $k$th quantile, is used to switch between a mode favoring uncertainty-based exploration ($\beta^\mathrm{explore}$) or semantic exploration ($\beta^\mathrm{go}$). As long as the agent is in an interesting state, with $\hat{r}^{sem}_t \geq Q_k$ , it keeps exploring with high uncertainty maximization ($\beta^\mathrm{explore}$). Only when the agent reaches a state that is not interesting anymore, indicated by $\hat{r}^{sem}_t < Q_k$, it more strongly favors exploration based on semantic rewards ($\beta^\mathrm{go}$).

Let’s consider the Robodesk setting as an example. A typical observation might have the robot just move the gripper across or over the table. However, when the gripper is near the ball on the table, this could be a more interesting state than what it typically encounters ($\hat{r}^{sem}_t \geq Q_k$). The agent would try actions that cause high uncertainty. When these actions lead to pushing the ball off the table, the agent ends up in an uninteresting state again with no object nearby to interact with ($\hat{r}^{sem}_t < Q_k$). Thus, it strives to mainly maximize semantic rewards again, for example by moving the gripper to the next available object.

We thank the reviewer for allowing us to further clarify this and we have now expanded our description of this mechanism in the main paper.

In Figure 5, why is there no comparison with VLM-MOTIF, whereas in Figure 4, there is?

We add this baseline now in Suppl. D5. We show that SENSEI outperforms VLM-Motif in most interactions metrics (Fig. 15) and rewards (Fig. 13), showcasing the importance of the information gain objective of SENSEI in the Robodesk environment as well.

Dear Reviewer dZbp, Could you please read the authors' rebuttal and give them feedback at your earliest convenience? Thanks. AC

We thank the reviewer for the excellent feedback. We gladly answer the open questions:

Do SENSEI’s improvements mainly come from VLM size increase?

It is true that Motif relies on a smaller LLM than GPT-4 used in our work. However, the improvements of SENSEI cannot only be attributed to scaling foundation models, but the way how semantic rewards are adaptively combined with uncertainty-based exploration. For example, our VLM-Motif baseline also uses GPT-4 annotations. Nonetheless it rarely discovers environment rewards during exploration (see Fig. 4). SENSEI, on the other hand, uses the semantic knowledge distilled from VLM annotations as a starting point for exploration and then branches out to discover new behavior. We also want to highlight that Motif in the original work is only text-based. Even though we have an increase in model size in our case, the difficulty of annotations is also significantly increased in our problem setting since our annotations are image-based, requiring spatial grounding capabilities from the VLM.

Training resources

Thank you for the great suggestion. We add a new section in which we detail all computational resources needed to train SENSEI in Supp. D.7. for all stages of SENSEI.

Smaller training data

This is a good point that we would like to share our insights on. Especially in environments with rich and high-dimensional observations with a potentially long-tailed distribution, such as in Robodesk, we believe more data is useful to make sure we don’t run into many out-of-distribution (OOD) states while running SENSEI. We see signs of this already in some of our experiments. For the right camera only ablation in Robodesk (Supp D.2), we keep the entire dataset with 200K samples. Since the annotations use a single camera angle, e.g. the drawer can be occluded, this can lead to false annotations from the VLM. That’s why in our main experiments, we annotate the same dataset of 200K pairs also with the left camera angle and only keep the pairs where both annotations agree. Doing so we only keep 69% of the original data, which is then used for Motif training. As a result, our annotations with dual-cameras are much less noisy, leading to improvements in drawer interactions compared to the right-camera only variant. However, we don’t yet match the performance of an oracle annotator, or even the right camera angle for some objects. We hypothesize this is also in-part due to the data we filter out in the process, increasing chances of OOD states and thus noise in semantic rewards during exploration with SENSEI. We expect however that the OOD issue to not be as significant in Minihack environments, where the observations are not as rich and diverse as Robodesk.

In order to decrease dataset sizes for Motif training without increasing OOD risk in Robodesk-like environments, we could try to ensure better coverage in the dataset used for reward model training. We believe this ties in nicely with our motivated future work, where we would use the buffer generated from a SENSEI run, that has richer interactions, to bootstrap a new Motif network.

We hope we could clarify all open questions and would like to thank the reviewer again for their feedback.

Dear Reviewer 5Hj6, Could you please read the authors' rebuttal and give them feedback at your earliest convenience? Thanks. AC

We thank the reviewer for their valuable feedback and great suggestions.

Coefficient adaptations & Hyperparameter sensitivity

Thank you for this excellent suggestion to ablate the coefficient adaptation strategy. In our new supplementary experiment in Robodesk, we test a version of our method with fixed reward weights (Supp D.6)

We present the results in Fig. 16 for six different sets of fixed weights. First of all, we observe that none of the fixed scale settings outperform SENSEI, nor do they consistently perform as well as SENSEI. Secondly, we see that the exploration behavior is very sensitive to the choice of these fixed weights. E.g., for larger weights on the semantic reward, the behavior collapses to mostly interacting with only specific objects.

In conjunction to this, we also tested the hyperparameter sensitivity of SENSEI when using dynamic scaling of the reward weights. We showcase in Figure 17 that across different hyperparameter configurations, SENSEI’s behavior is much more robust and is better or at least on par with Plan2Explore in all cases. We don't observe any behavior collapse, in contrast to the fixed scale setting. We would therefore argue that the overall behavior of the dynamic scaling is much more robust and less dependent on hyperparameter tuning compared to fixed reward coefficients.

Exploration baselines

Thank you for your baseline suggestions. We have added a new exploration baseline to further demonstrate the advantage of using SENSEI. In Robodesk we now also compare SENSEI to Random Network Distillation (RND) [ref 1], a popular exploration strategy that uses the prediction errors of random embeddings of input images as an intrinsic reward to guide a policy towards unseen regions. Except for button presses, SENSEI interacts with all objects more frequently than RND and, as a result, discovers more rewards during exploration.

Regarding the other suggested baselines – Unfortunately we do not think these are particularly suited or applicable for the setup we consider. As discussed in Section 3, ELLM and OMNI require text-based environments or a mapping from environment states to text, where OMNI also assumes access to reward functions for all potential tasks in the environment during training. On the other hand, SENSEI is designed for environments with pixel-based inputs without assuming access to environment task rewards during exploration.

LEXA is a goal-conditioned extension of Plan2Explore: after an exploration phase with Plan2Explore, in a second stage, LEXA randomly samples goals from its replay buffer to train a goal-conditioned policy. While applicable in our environments, the exploration phase in LEXA is Plan2Explore. We see LEXA’s addition of goal-conditioned RL to the Dreamer framework as an orthogonal research direction to our work that could also be applied to SENSEI. For example, Plan2Explore exploration phases in LEXA could be replaced by exploration with SENSEI, and the uniform sampling from the replay buffer can be replaced by sampling from the top-k samples, as ranked by our VLM-Motif reward.

PEG extends LEXA by a more sophisticated exploration by searching for exploration goals likely to lead to high epistemic uncertainty. Unfortunately, PEG performs its goal search in observation space. Thus, it requires lower dimensional observations (positions) and is not applicable to our pixel-based environments.

LAMP proposes a pre-training strategy for a language-conditioned policy, given a diverse set of tasks generated by hand or by an LLM. This means that e.g. for Robodesk, LAMP in the pre-training phase already tries to optimize for tasks such as “open the drawer”, “lift the block”. Compared to our work, the focus is not on discovering useful behaviors to engage with in a given environment, but on trying to learn a given set of useful behaviors better. Given this discrepancy in the method’s objectives and experimental setup, it is unclear to us how to ensure a fair comparison between them.

Hyperparameter sensitivity

As mentioned above, we overall found the behavior of SENSEI to be robust to hyperparameters, and did not observe behavior collapse in any of the settings we tested for. As a general rule, we did perform a hyperparameter grid search for all of our experiments (SENSEI and all baselines), and presented the best results we obtained in each case, maintaining a fair optimization budget over all tested methods.

Seeds

As per your suggestion, we now increased the number of random seeds to 5 in all Minihack experiments. For the final version of our paper, we plan to further increase the number of seeds and also add more seeds to our Robodesk experiments.

References

[ref 1] Burda, Yuri, et al. "Exploration by random network distillation.", ICLR 2019.

Dear Reviewer 71Ub, Could you please read the authors' rebuttal and give them feedback at your earliest convenience? Thanks. AC

Thank you for the feedback and raising some important questions.

Criticism on human bias of interestingness for environments

I believe environments where humans can clearly express preferences between observations based on interestingness are limited

We agree that for abstract environments, such as simple mazes or abstract games, it is unclear whether a pretrained VLM could extract a human bias of interestingness from screenshots alone. However, our long-term goal with SENSEI is to tackle realistic scenarios, such as real-life robots or highly realistic simulations or games. We believe that photorealism of observations are likely to help VLM annotations because a large portion of their training data comes from real-world photos or videos. For example, without providing an explicit context to a pretrained VLM, a camera image of a robot holding an object could probably be easier to interpret than a screenshot from gridworld environments or a Go board.

While it’s true that in the abstract and simplified environments currently used for RL research, SENSEI might not always have an edge, we believe that as environments become richer and VLMs become more powerful there will be more and more potential applications for SENSEI.

We thank the reviewer for raising this point and we now emphasize this more clearly in our discussion.

Task specification in prompts

In my opinion, the actual prompt the authors use is simply a more specified version of that used in [1]

We believe there might be some confusion regarding the types of prompts used in the Motif paper [ref 1]: Motif also uses environment specific knowledge to reduce noise in LLM annotations. Motif is only tested in Nethack and as default uses what the authors call “modifiers” in their prompts. In almost all of their experiments, the modifier contains environment specific knowledge, where the default modifier is:

"Prefer agents that maximize the score in the game, for instance by killing monsters, collecting gold or going down the stairs in the dungeon."

On top of that, Motif is uniquely situated as it is tested on NetHack: There is an abundance of publicly available NetHack wikis, guides etc. on the internet, which are likely also part of the LLM training data. Thus, Motif relies on the fact that the LLM is likely familiar with the game, which the authors also highlight in their paper.

However, we do agree that the Robodesk prompt in our main experiments is likely over-specified and we could use less explicit, zero-knowledge prompts to train SENSEI. We illustrate this in a new experiment in Suppl D.4. Here, we annotated images using a multi-turn prompting strategy without prior information about the environment (details in Supp. C.3.2). First, we show a picture from the robotic environment of Robodesk, and ask the VLM for an environment description. Next, using this description in-context, we prompt the VLM to annotate pairs of images based on their interestingness. We analyze the reward function distilled from this general prompting strategy and show that it behaves qualitatively very similar to our original reward function, which used more environment-specific prompts, and matches the peaks in “interesting” moments of interactions.

The reason we opted for a specialized prompt in our initial experiments is two-fold: First, similar to Motif, we wanted to increase annotation accuracy. Especially, since we are dealing with images that are not necessarily photorealistic and potentially out-of-distribution compared to the type of data VLMs are trained on. However, the second reason, which was our main concern, was due to practical limitations: Zero-knowledge prompts with multi-turn dialogues using GPT-4o cost double per annotation. With our specialized prompt, using a single-turn strategy, we were able to annotate 200K pairs for $400. For this new experiment, however, we could only annotate 100K pairs with this budget. As open-sourced VLMs get better, we expect multi-turn zero-knowledge strategies to prevail without cost constraints.

Dear Reviewer qRFT, Could you please read the authors' rebuttal and give them feedback at your earliest convenience? Thanks. AC

We thank all the reviewers for responding to our review and providing more feedback. To address concerns expressed in some of the responses, we provide two important updates to our paper.

For the reviewer’s convenience, we highlight these newest changes of the paper in pink, while previous changes are highlighted in red.

SENSEI with image annotations from general, zero-knowledge prompt

In our first revision, we were able to distill a reward function using a more general prompting strategy without external knowledge about the environment by first prompting the VLM for an environment description and using this context for annotation (details in Suppl. C.3.2).

We now run SENSEI with this general prompting setup in Robodesk. In Fig. 5 we show that this more general version of SENSEI without external knowledge interacts roughly as often with relevant objects as our original SENSEI with an environment description generated by us. Thus, even without external knowledge SENSEI clearly outperforms Plan2Explore and RND in overall number of object interactions. We believe this demonstrates how SENSEI’s performance does not hinge on specific prompts and these new results showcase the generality of our approach.

Even more seeds

We were able to increase the number of random seeds of our experiments. We now show MiniHack interactions with 10 random seeds per configuration in Fig. 4. As expected, the previous trends hold and SENSEI outperforms the baselines in the number of rewards obtained, now also with reduced variance.

We are currently rerunning all experiments of the main paper to increase the number of random seeds to 10 for all SENSEI configurations and baselines. Since we cannot modify the paper further, we will upload the updated plots (Fig. 5 & Fig. 6) to our supplementary website (linked in the paper, first footnote) once they are finished.

We hope these changes address any remaining concerns about empirical evidence. We are happy to answer further questions.