We thank the reviewer for their feedback!

Significance of method

it appears more as a choice of system design to generate heuristics rather than a fundamental method

We respectfully disagree with the reviewer that, because we are demonstrating a capability of applying language priors to imitation learning, our contribution is not “a fundamental method”.

First, as we know from the recent boom of LM+robotics papers, the premise of applying strong pretrained language priors to problems of decision-making is of high value and immensely impactful to the field. We simply take an alternative approach from those of previous methods: rather than leveraging LMs for action abstraction i.e. action stitching as in SayCan, we use them for state abstraction, i.e. identifying relevant features. LGA therefore demonstrates the utility of strong LM priors in a novel facet of decision-making.

Second, we agree that VIOLA is an important related work and will update our paper to discuss it in more detail. However, it is not evident to us that learning important state features should ideally be learned with the motion policy: indeed, this forces the agent to re-learn motor policies for novel tasks (e.g. a new linguistic specification), whereas our approach allows the re-use of motor policies across tasks (if the new linguistic utterance has been found to be the same as previously learned task with an alternative specification). However, we believe that learning how to blend these different state representations (both abstract and raw) is an important direction for future work, and will emphasize in the discussion.

Finally, we evaluate LGA with BC here since it is the clearest, simplest method for isolating the impact of good abstractions, but it is entirely feasible to utilize the abstraction generation process of LGA in tandem with a method like VIOLA for jointly learning well-attended policies (e.g. by also conditioning the transformer-based policy in VIOLA on LGA-generated abstractions). We are happy to clarify this scoping of LGA in the new revision.

Evaluation

VIMA is too simple

We appreciate the reviewer for pointing out the limitations of VIMA as an evaluation domain, and for the suggestion of RoboSuite. Although evaluating on harder benchmarks is a great suggestion for future work, we felt it prudent to instead spend time on demonstrating the practical feasibility of LGA on a real robot. As we wrote about in Sec. 7, we demonstrated the full LGA pipeline on a Spot robot, including using raw robot observations segmented via SAM, captioned via Dedic, and then imitation learned on real user tele-op’d trajectories. Consistently (~9/10 test tasks), the state abstractions produced via LGA were able to produce policies successful at completing the tasks. The few failure cases observed were due to the captioning model failing (e.g. SAM reliably detected different shapes and sizes of water bottles but Dedic was unable to always produce the text description “water bottle”).

Thus, the LGA pipeline can be reliably deployed in real-world scenarios, and we expect that it can be straightforwardly generalized to even more complex ones, especially as individual components of the pipeline improve. While we can always evaluate on harder/alternative simulated benchmarks, we hope the reviewer appreciates the effort spent on ensuring the full pipeline is able to be practically implemented with real hardware!

Question

do you use the binary mask directly (as visualized in Figure 1) or the masked image?

We used the binary mask directly, although the reviewer suggests a reasonable alternative. In our approach, we explicitly highlighted the flexibility of LGA to operate on both the binary mask as well as one conditioned on the raw observation, although note we did not include masked image as an alternative. We suspect that for tasks where texture/color is not meaningful, e.g. once the system has detected it’s an image, there’s no need to preserve the color “orange” in the observation”, although either would likely work. We’re happy to clarify this additional option in the revision.

Missing Literature

We thank the reviewer for bringing missing references to our attention, for they indeed help frame the historical context of state abstraction in decision-making! We will add them to the related work.

Dear Reviewer 4L8o,

Thank you for your review and feedback!

As the discussion period is coming to a close, please let us know if our response has adequately addressed your concerns, or if you have any remaining questions. If not, we would appreciate if you could raise your score. Thanks for your work in serving as a reviewer!

Thanks, Authors

We thank the reviewer for their feedback!

Questions

Does LGA assume full observability or run segmentation on every frame?

In Sec. 7, we explored the feasibility of implementing the full LGA pipeline on a real robotic system: a Spot robot with arm, real segmentation via SAM, and real captioning with Dedic. In order to avoid the real-time latency of running SAM+LM on every timeframe (as the reviewer suggests), we did assume full observability of the scene at the initial timestep, which is a common assumption many vision-based robotics works also make (e.g. the robot needs to see objects in order to manipulate them). However, if/when running inference of such models becomes less expensive, it’s certainly feasible to apply LGA at every timestep, or every k timesteps as needed!

How well does [segmentation and captioning] work in the real world? What are common failure modes?

As noted above, real-world segmentation + captioning worked quite well on the real robot, reliably able to create good state abstractions successfully ~9/10 times for each scenario (bringing different kinds of fruit and throwing away different types of cans/bottles). The few failure cases observed were due to the captioning model failing (e.g. SAM reliably detected different shapes and sizes of water bottles but Dedic was unable to always produce the text description “water bottle”). As a result, we are quite hopeful that the LGA pipeline can be extended to additional real-world scenarios, especially as individual components of the pipeline improve.

How good is the LM at guessing relevant objects not explicitly mentioned in the language instruction?

The reviewer is correct that potential task-relevant features must be referenced in the user’s language utterance (e.g. “pick up the fruit while avoiding obstacles” if object-avoidance is sometimes important to the task, sometimes not). However, LGA does afford a degree of flexibility not captured by prior methods, i.e. LGA does not require objects to be mentioned by name. While we did not test this directly, we did observe instances of this: for example, LGA was able to convert “something I can put food in'' to the object "bowl" and "ripe fruit” to “yellow banana” but not “green banana”. We will add these to the appendix and update the discussion to include this as an exciting future direction!

Significance of approach

Conceptually, the idea of using visual masks as attention or part of state representation isn’t novel and has been explored in various prior works including recent ones such as robotmoo[1] and VIOLA[2]

We agree that the idea of using visual masks as state representation is not new, and did not intend to claim it as a contribution. LGA demonstrates the feasibility of leveraging language priors to create these state abstractions. This is distinct from VIOLA, which explores the feasibility of leveraging image priors.

While MOO takes a step closer to our approach by leveraging vision-language priors, we note that first, the paper was arxiv’d a month after the ICLR deadline (and therefore is concurrent work to ours) and second, that their approach relies on models that perform caption-matching from datasets already grounded in real-world objects (as opposed to ours, which leverages language-only priors to flexibly map ANY language-specified feature into the scene, not just the goal object).

language instruction can be about a group of objects, the object as a whole or only part of an object and it is unclear how to segment or group segmentations before we know the task.

We certainly agree that these are important scenarios! Segmentation should ideally be fine-grained, as the LM can only be queried about segmented and captioned features. However, given fine-grained segmentation on the level of object(s), LGA can certainly handle situations such as “a group of objects”. For example, consider a task like “bring me all fruits from the table”, where both a tomato and a banana are present in the scene. We verified through a quick LM experiment that the abstraction created by the LGA would include both the tomato and the banana since the LM is queried about whether each individual feature in the scene is relevant to the specified task! We believe it’s this flexibility that differentiates LGA from VLM or alternative attention-based approaches for imitation learning, since the LM can flexibly construct ANY abstraction from text, not just for specific skills like “pick up [x] object”.

We thank the reviewer for their feedback!

Reviewer Question

Can you categorize task failures into two groups: those caused by policy quality and those resulting from incorrect state abstraction?

The experiments we ran were intended to isolate the effect of good abstractions generated via LGA vs. bad abstractions (i.e. raw observations containing spurious features). Consider the tasks shown in Fig. 4. For each task, we fixed the training and test distribution for demonstrations for all LGA variants and the GCBC baseline. In other words, we held the training data, test data, and model training constant, varying only whether the model was trained on the LM-generated abstraction vs. the raw observation (which intuitively is a bad “abstraction” containing many spurious features). The resulting performance gap can therefore be attributed to the quality of the abstractions used for learning. Our hypothesis is that bad abstractions (containing irrelevant features) will perform poorly when confronted with covariate shifts such as new colors or distractors. We note this evaluation is similar to work in reward learning, which often assesses the quality of the learned reward via policy success on the downstream task.

That said, we agree with the reviewer’s point that the current results do not explicitly specify the correctness of abstraction generation. To verify that our evaluation above does indeed capture the effect we claim, we have defined an additional metric assessing similarity of the LM generated and human-designed feature sets. We measure the Hamming distance, where smaller values indicate similarity. As an example, for the four tasks in Fig. 4A, the following results show that 75% of the time, LGA generated feature sets match those of human engineered sets when compared to those from raw observations.

Iterative Abstraction Construction

whether refining its choices with feedback from policy execution is a feasible solution remains unexplored.

We think this is an excellent suggestion, and one that we are currently exploring in future work! Notably, for this type of iterative abstraction update to be feasible, one must assume a way of generating rewards/scores for judging policy rollouts. We did not feel iterative updates were central to the idea of LGA (which explored the feasibility of leveraging LM priors for constructing state abstractions, not for scoring policy rollouts), so we decided to separate out into future work. We are happy to discuss this choice in the discussion!

Missing literature

The paper appears to overlook extensive research on learning state abstraction for reinforcement learning

Thank you for highlighting these relevant and important works. We note that 2 were already cited in the paper (see the two Abel et al. papers on p. 1 and 3). We agree the Abel et al, 2019 paper is also relevant: briefly, LGA differs because it constructs such abstractions via language-guided priors. We will update the paper to incorporate and discuss this work.

I remain positive after reading the rebuttal. The paper introduces a simple approach to leverage LLM for effective robot learning by state abstraction. This method sets it apart from existing approaches, which primarily use LLM for planning and reward learning.