Behavioral Exploration: Learning to Explore via In-Context Adaptation

We thank the reviewer for their feedback. We have added videos illustrating the performance of our approach and baselines on our real-world WidowX setting here, and provide clarification on additional points below.

The claim in the introduction is that existing works are "slow and expensive”…

It is well-known that randomized exploration, on which the majority of practical RL approaches are based, is “expensive” in the sense that it requires many interactions before it discovers the correct behavior (see e.g. [1]). Furthermore, most RL algorithms update their behavior “slowly” by gradually fitting a value function to newly collected data, and updating their policy to maximize this value function. While this approach can fit the data after sufficiently many gradient steps, to avoid training instability one must typically limit the number of gradient steps per sample, causing the policy to lag behind the behavior that could in principle be learned from the data (see e.g. [2]). We believe these statements are backed up by our experiments (Figures 2 and 3, in particular), which illustrate that our approach of in-context adaptation leads to much faster exploration and task-solving ability than RL approaches, and furthermore these observations are common throughout the RL literature.

The introduction example claims that the robots should not explore…

While we concur that in certain settings—in particular, when very little prior knowledge is available—taking “wrong” actions is indeed useful, in settings where one already knows an action is “wrong” taking this action is of limited usefulness. For example, if the agent knows what a “cup” is, then it already knows that picking up a plate is incorrect, so attempting to pick up the plate does not help it learn anything new about which cup it should pick up. Furthermore, as there are many possible "wrong" behaviors, exploring them all is very sample inefficient.

The focus of this paper is on scenarios where the prior knowledge (in our case, in the expert demonstrations) is rich enough to avoid many “wrong” behaviors, and where we want to quickly explore over potentially correct behaviors. We believe this is useful in many real-world settings. For example, ideally we would want a robot policy that follows instructions (when we tell it to “pick up the pot” it picks up a pot, and not some other object) but that can try different behaviors until it picks up the correct pot. Our experiments illustrate that OpenVLA, a state-of-the-art generalist robot policy, can do the former yet not the latter, and existing work on exploration can achieve the latter (after many samples attempting every possible behavior) but not the former—our work fills a gap in the literature by enabling a policy to achieve both simultaneously.

D4RL is a relatively new benchmark…

We would like to emphasize that D4RL is nearly 5 years old and has become the standard benchmark for nearly all work in offline RL since its release (see e.g. [3]-[5]). Furthermore, other RL benchmarks (such as Atari) do not have standard offline datasets—which is critical in our setting—and are therefore not amenable to this work.

Variances are not shown…

Figures 2-4 do include error bars—see, for example, the BC curve of Figure 2. As stated in the main text, these error bars denote 1 standard error.

Tabular results…

We will include tabular results in final version of the paper, as requested.

Consistent results in Figures 2-4

Figures 2 and 3 consider different metrics—Figure 2 plots the number of goals reached, while Figure 3 the total number of regions reached—and Figure 4 a different environment entirely, and so we would not necessarily expect the ordering to be the same.

Measuring coverage…

As noted in footnote 1 in our paper, measuring coverage through the inverse feature matrix is a canonical technique in statistics and, furthermore, is common in the RL theory literature (e.g. [6]-[8]).

[1] Dann, Chris, et al. "Guarantees for epsilon-greedy reinforcement learning with function approximation." ICML, 2022.

[2] Li, Qiyang, et al. "Efficient deep reinforcement learning requires regulating overfitting." ICLR, 2023.

[3] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." NeurIPS, 2021.

[4] Kostrikov, Ilya, et al. "Offline reinforcement learning with implicit q-learning." arXiv, 2021.

[5] Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning." NeurIPS, 2020.

[6] Jin, Chi, et al. "Provably efficient reinforcement learning with linear function approximation." COLT, 2020.

[7] Wagenmaker, Andrew J., et al. "Reward-free rl is no harder than reward-aware rl in linear markov decision processes." ICML, 2022.

[8] Zhou, Dongruo, et al. "Nearly minimax optimal reinforcement learning for linear mixture markov decision processes." COLT, 2021.

We thank the reviewer for their feedback. We have added an additional unsupervised RL baseline to the Libero experiment, which we found performed worse than both BE and BC, and have also run additional experiments with OpenVLA, testing out the reviewer’s suggestion for enabling OpenVLA to explore. See below for discussion on these points and additional clarifications.

[Concern 2] New RL Baseline on Libero

We have added SUPE, the most effective unsupervised RL baseline, on Libero. Please see Figure 1 here. As illustrated, SUPE performs significantly worse than both BC and our approach. We note that the number of episodes we consider for Libero (15) is significantly less than what is typically required by RL algorithms to effectively learn, so we do not find this result surprising—it further illustrates the necessity for fast in-context adaptation as compared to standard RL gradient-based updates.

[Concern 4] New OpenVLA Experiments

We make several comments on our inclusion of OpenVLA as a baseline. First, our goal in including it is to demonstrate that existing state-of-the-art methods for robotic control do not effectively explore over the space of possible tasks when the goal specification is ambiguous. In other words, OpenVLA does not already solve the problem we are attempting to solve—if the specified goal is ambiguous, OpenVLA does not attempt different behaviors until it finds the correct one, it simply chooses the behavior it deems most likely.

Second, as the reviewer has highlighted, OpenVLA receives no feedback on whether it has succeeded. This is fundamental to OpenVLA, however, as well as most other current state-of-the-art BC methods for robotic control: OpenVLA is not trained to condition on a history of past interactions or observations, and therefore fundamentally cannot adapt online. Our approach aims to fill precisely this gap, and we hope it will be incorporated in future policy training to enable effective adaptation to history. We note as well that, as discussed in our reply to Reviewer kepy and illustrated by our new BC with history conditioning baseline on Libero, naively conditioning on history can actually hurt performance—our approach may be a way to instead improve performance when conditioning on history.

Given this, instructing OpenVLA to pick up “another” object is an ambiguous command—since OpenVLA has no history dependence, it has no grounding for what “another” object is. Nonetheless, we ran an additional trial with OpenVLA on the command “pick up the other silver pot”. We evaluate on the task illustrated in Figure 18, and give the results below, indicating which fraction of episodes the policy goes to the pot on the left, the right, or fails to grasp at either, and compare to our original command.

As this illustrates, while this command does cause OpenVLA to move to the left pot somewhat more frequently, it also hurts its ability to move to either pot at all (likely because this command is somewhat outside the training distribution for OpenVLA).

[Concern 1] Ease of Training and Evidence for Faster Adaptation

To clarify our claim that the method is easier to train, this is indeed somewhat tangential to the claim of faster exploration and adaptation online, the primary claim of our paper and the main focus of our experiments. However, our approach does lead to lower computational overhead and simplicity at deployment compared to the baseline RL methods, all of which require online gradient updates and hyperparameter tuning. While this is an advantage of our approach, we see this as complementary to our main objective.

We would also like to emphasize that BE is able to explore significantly faster on Antmaze and Libero than all RL baselines, supporting our main claim of faster adaptation and exploration. For example, to reach the final performance of SUPE, the best RL baseline, it requires only 15% of the samples. While the performance between BE and the baseline RL methods is similar on Franka Kitchen, we believe the strong performance on Antmaze and Libero highlights that in many settings, our approach leads to a substantial gain.

[Concern 3] Scene Conditioning

As noted in the paper, Libero 90 consists of 90 tasks distributed across 21 scenes. The scene conditioning vector is then simply a one-hot vector of dimension 21 indicating which scene the agent is operating in. While this information can be inferred entirely from visual information (the scenes appear visually distinct) we found that adding this conditioning improved the performance of both our approach and BC slightly. This is not critical to performance, however.

We thank the reviewer for their feedback. We have run an additional requested baseline on Libero—BC with history conditioning, which we found performed worse than BC—and additional experiments illustrating that BE adapts to its history online; please see below for further discussion and the results here.

New Experiment Showcasing BE’s Ability to Adapt Online

Please see this Figures 2 and 3 here for illustration of the BE’s ability to adapt to history. To illustrate adaptivity, we run a BE policy trained on Antmaze Large, conditioning on either (a) history sampled from states visited in past online episodes (the strategy used in our experiments), or (b) histories containing only a single fixed trajectory. As our results illustrate, the history can have significant impact on the agent’s performance: when conditioning on a single trajectory, the agent largely takes routes that avoid the space traversed by this trajectory, adapting its behavior to visit novel states. Given this, we see that the agent achieves significantly higher coverage conditioning on the past states it has observed, rather than a single fixed trajectory.

New Experiment Showing History Conditioning Hurts Standard BC

We have a trained a BC policy with history conditioning on Libero—please see Figure 1here. As can be seen, conditioning the BC policy on history actually hurts performance. We believe this is an example of “causal confusion” [1], a known phenomenon where conditioning on additional information can decrease performance of BC by introducing spurious correlations.

Learning on Downstream Tasks

The reviewer states that: “A key claim is that BE learns... The experiments... downstream tasks”. We want to highlight that the main focus of the paper is not training on downstream tasks, but rather pretraining a policy that can explore effectively online, focusing its exploration on behaviors exhibited by the demonstration data. While existing work (e.g. [2]) has shown that a richer supply of data does improve policy learning on downstream tasks—and given this we believe that data collected with BE may indeed be useful for learning downstream tasks—this is not the main focus of the paper. We are instead primarily interested in how to pretrain a policy for exploration, and our experiments were chosen to reflect this. We will update the exposition in the paper to make this clear.

Additional Clarifications

History Distribution: At deployment, we condition the BE policy on the history of states it has already collected online. As any policy $\pi$ induces some distribution over states visited, the distribution of history encountered at test time depends on the learned BE policy. Ideally, we would like the distribution the BE policy was trained on to match this online distribution. As described in Section 4.3, this is challenging as we do not know what the state visitation distribution of the learned policy will be. Addressing this is a fixed-point problem in that we must first optimize a BE policy on some history distribution, deploy it to find what history distribution it induces, then re-optimize a policy on this new history distribution, and repeat until convergence. To avoid this complication, we suggest choosing the history distribution in training to be a uniform distribution over trajectories in the offline dataset, which we found worked well in practice.

Number of Demonstrators: The offline data may be collected by different demonstrators; in this case $\pi_{\beta}$ is a mixture policy over all demonstrator policies.

How can we decouple the performance improvement…?: While our approach can learn to explore effectively with any sufficiently diverse set of offline data, it focuses its exploration on the behaviors exhibited in the offline data. Therefore, while diverse low-quality data may be sufficient for training a policy that collects diverse data, it is not necessarily sufficient for successfully achieving a task; the demonstration data must contain examples of successful task completion for BE to learn these behaviors. Thus, both the learning to explore and the type of demonstration data are important for our approach.

How is the coverage objective…: Would the reviewer be able to provide additional clarification for what they mean by “feature matching” here? One potential difference is that BE aims to induce different behaviors, while typical BC usually aims to induce the same behaviors.

[1] De Haan, Pim, et al. "Causal confusion in imitation learning." NeurIPS, 2019.

[2] Yarats, Denis, et al. "Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning." arXiv preprint arXiv:2201.13425 (2022).

We thank the reviewer for their comments. In the following we provide additional clarification on the success metrics and RL experiments, as well as several other comments the reviewer had. We will update the final version of the paper to include all these details.

Clarification of Success Metrics

For the real-world WidowX experiments, we define an “interaction” with an object as the end-effector of the WidowX making contact with the object (as stated in the supplemental: “We consider a reach to be successful if the end-effector makes contact with the object”). We note that in the settings we considered this is straightforward and unambiguous to evaluate. As we are evaluating the ability of each approach to explore the possible tasks in the scene, we run the policy 5 times, and count the overall sequence as a success if across these 5 trials both objects in the scene were interacted with—the end-effector made contact with each object at least once across the 5 episodes.

For Libero and D4RL Kitchen, we utilize the built-in success detector in the simulator to identify success. For D4RL Antmaze, we utilize the same goal locations as in (Wilcoxson et al., 2024), and count a goal as reached if the agent reaches within distance 0.5 of a square of side length 1.5 centered at the goal point. For each setting, we evaluate on each task separately (e.g. each goal for Antmaze, or task for Libero), and the plots given in the paper are performance averaged across each task (see below for further clarification on aggregation of results). Figures 2, 4, 6, 7 therefore show what fraction of trials have resulted in a success up to a particular time (averaged across tasks and random seeds).

We will add all these details to the final version of the paper. If the reviewer has any additional questions on the success metrics we are happy to provide further clarification.

Clarification on RL Problem Setting and Experimental Details

For all RL baselines, we run them as described in their original works. In particular, for each setting the algorithm is given an offline dataset of transitions from the environment, but with no reward labels. With the exception of “Online RND”, each method we consider pretrains on this data (Online RND only trains online). After pretraining, each method is deployed online, and is evaluated on the number of steps it takes to reach the specified goal at least once.

The BE policy is therefore used only for exploration, and is not fine-tuned at all during online deployment. However, as described in Section 4, the history of previous (online) interactions is fed into the context of the BE policy online, so that its behavior can adapt to the past experience—this adaption is only “in-context”, however; no gradient steps are taken on the BE policy. For every other approach considered, with the exception of BC (which is not updated at all online), the policy is updated online with standard gradient-based RL updates (in particular, which seek to maximize RND bonuses, inducing exploration).

Aggregated Performance Curves

We observe similar trends in the individual results as in the aggregated results, but for the final version will include in the upplemental individualized results on Antmaze (for each maze type and goal location) and Libero (for each of the 90 tasks). For the WidowX results, the supplemental already provides individualized results (see Figures 20-22). In all cases where results were aggregated (Antmaze, Libero, and WidowX settings), the aggregated results are the mean of the individual results.

Fine-Grained Control

In addition to Antmaze, which, as noted by the reviewer, does require joint-space control, the Franka Kitchen environment considered in Figure 4 also requires joint-space control. We believe that these results indicate that our method does scale effectively to settings with joint-space control.

More generally, we would expect our method to scale effectively to any setting on which BC performs well. As our method is essentially training on a BC objective augmented with a particular conditioning structure, we would expect it to, in general, perform at least as well as BC, which we found to be the case in all our experiments. The degree to which it learns effective exploratory behavior is a function of several additional factors, in particular the distribution of behaviors present in the offline data, so there may be cases where it is not able to learn more effective exploratory behavior than BC—we do not necessarily expect the type of control to affect this, however.