Flex: End-to-End Text-Instructed Visual Navigation with Foundation Models

We sincerely thank the reviewer for their dedicated and meticulous evaluation of our paper. Their thoughtful insights and careful reading have been pivotal in refining our work. We have diligently addressed each of their valuable concerns and are eager to engage in further discussions to resolve any remaining issues.

Weakness 1

The reviewer makes a good point in inscribing the work in the larger scheme of robot visual navigation. We argue that the admittedly simple task does not limit the significance of the findings and does not contradict the quoted assertion.

Platform-Agnostic Pipeline

First, the entire pipeline is completely platform agnostic, except for specifying the action space. The 4-dimensional action space used here is, in fact, more complex than the case of driving (often 2D with steering angle and acceleration). We know from related work that the pipeline presented will translate seamlessly (see [1] as an example). This also applies to humanoids and quadrupeds, at least at the higher level of abstraction of actions learned.

Simplicity and Future Work

Second, we agree with the reviewer regarding the simplicity and uniqueness of the task exhibited. This is, in fact, acknowledged in the limitations section A2 of the appendix. Aspects that are left for future work include the inclusion of multiple behaviors that go beyond reaching landmarks (such as obstacle avoidance, direction following, etc.), which will go hand in hand with the inclusion of memory. This will enable Flex to serve as the base architecture for complex robot navigation tasks across multiple domains.

Generalization and Controlled Experiments

Nevertheless, the goal we set with this project is to identify the ingredients necessary for generalization in both environments and goal targets. We hope the reviewer will appreciate that a single controlled experiment—with ablations and analysis in terms of data collection, patch feature resolution, and policy network choice—provides a unified playground for clear comparisons, analysis, and conclusions.

In this context, we do identify a solution that is able to achieve the specified task in the real world from simple simulation data, with open dictionary goals specified via text by a user. It is in that sense that our generalization claim holds.

Reference

• [1] "Drive Anywhere: Generalizable End-to-End Autonomous Driving with Multi-modal Foundation Models" by Tsun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Yutong Ban, Alexander Amini, Guy Rosman, Sertac Karaman, and Daniela Rus.

Weakness 2

We thank the reviewer for this very insightful question, which sheds light on important differences between Vision-Language Models (VLMs) and their capabilities.

Embedding vs. Fusion of Image and Text

Both CLIP and ImageBind can embed pairs of images and text together, but not directly in the sense of fusing the two modalities into a single embedding. Instead, these models independently process images and text into separate embeddings that exist in a shared latent space. Thus, there is no direct interaction or integration between the modalities during the embedding process. Their embeddings are simply aligned in the shared space. This approach is well-suited for tasks like retrieval, similarity measurement, and zero-shot learning across modalities but does not produce a joint representation of paired inputs.

Direct Fusion in BLIP-2

On the other hand, BLIP-2 integrates image and text inputs directly, allowing the model to process the interactions between the two. This fusion enables a deeper understanding and reasoning about the relationship between the image and text, which is essential for tasks requiring combined context, such as robotic navigation from instructions.

Selection of Models

We conscientiously avoid models that do not produce a unified fused representation from an image and text pair. Other models that offer such feature representations—like LXMERT, ViLBERT, and UNITER—share similar architectures. We have thus proceeded to select the most popular models in use to validate our approach.

We are happy to discuss with the reviewer the possibility of extending our analysis to specific feature extraction models for the camera-ready version of the work.

Weakness 4

The reviewer brings up a good point that a more difficult formulation of a navigation task would be to navigate to a precise distance away from the target object.

Clarification of Task Evaluation

We would like to highlight to the reviewer that, as described in Section 4.1, the evaluation focuses on whether “the agent can navigate towards the user-instructed object and center it in the middle of the frame.” While this does not explicitly enforce navigation to a precise distance, it inherently involves assessing the agent’s ability to maintain an appropriate relative positioning with respect to the target. Moreover, we test the model across diverse conditions, including:

• Out-of-distribution visual backgrounds
• Target objects of varying sizes
• Different instructions
• Simulation-to-real transfer

This demonstrates robustness in meeting this task specification.

Data Type and Framework Consideration

More importantly, we do not believe the question here is about the amount of training data used, but rather the type and modalities of data considered for a specific framework. In this work, we operate within the visual (and visual-only) end-to-end control learning setup. The rationale is that this is a popular approach applicable to a wide array of robotics platforms and applications, avoiding problem overspecification. It is also a framework that readily reaps the progress in computer vision and multimodal foundation models enabled by the huge amounts of image data available on the internet (which we explore in this work).

Blueprint for Future Tasks

Thus, we argue that although our selected task is rather simple and perhaps ill-specified in the sense of accurate distance monitoring, it serves as a blueprint approach that can be extended (albeit with further improvements and development) to other visual navigation tasks across various platforms (e.g., ground and sea vehicles, humanoids, quadrupeds, etc.) in a sensor-agnostic fashion.

Question 1

We thank the reviewer for these comments, and we believe they highlight an important point. Below, we provide a clear description of the experimental setup and results. We are happy to emphasize these details more prominently in the paper following the reviewer’s insightful suggestions.

Environment

Our policy model was trained in a single simulated environment called Samurai. At test time, the model was evaluated across three distinct scenarios:

1. Same Training Environment (Samurai)
2. A New Simulated Environment (Stadium)
3. The Real World
   • Tests were conducted in various and diverse surroundings.

Objects

• Training Data:

  • Training utilized only two spherical objects with specific colors: blue and red.
  • These were the only objects present in the training data.

• Generalization Performance:
  The model demonstrated significant generalization to open-set scenarios:

  1. Mixed Color Shapes (Simulation):
     • Generalized to objects with varying shapes and colors, beyond those seen during training.
  2. Open Dictionary (Simulation):
     • Tested on a wide range of simulated objects, including:
       • A blue sphere
       • A light-colored Jeep
       • An Australian cattle dog
       • A brown horse
       • A tall and narrow palm tree
       • A toy space rocket
       • A whole watermelon
  3. Open Dictionary (Real World):
     • Evaluated on various real-world objects during drone navigation, including:
       • A yellow star
       • A bicycle
       • A man with a wig
       • And more.

We are happy to emphasize these points more prominently in the paper based on the reviewer’s valuable feedback.

We sincerely thank the reviewer for their professional evaluation and detailed feedback. We look forward to further engaging during the open discussion period. To initiate this dialogue, we have provided comprehensive responses to all the issues and questions raised in your initial review, hoping it will encourage you to increase your score.

Weakness 1

We agree with the reviewer and thank them for contributing to an improvement in the presentation. In the revised version of the manuscript, we have moved some of the related work citations from the introduction to the related works section and merged the Introduction with the Preliminaries.

Thanks for the careful reading.

Weakness 2

The reviewer makes an important and valid point. We provide some nuance as to its pertinence to our approach.

First, we note that while the number of image-action pairs in our dataset is slightly large (as this is a regression task), the number of demonstrations is considerably small, totaling 300. Furthermore, the training dataset includes only 1 or 2 distinct objects, making the diversity of training data even more limited. Additionally, our training is conducted in just a single environment. Despite these constraints, we demonstrate open-set generalization across text, objects, and environments. Compared to previous works, this is the first study to achieve such robust generalization from a limited set of environments, objects, and demonstrations.

Second, our argument would be that this is not an open-world navigation paper in its widely adopted sense. The emphasis in this work is not to train a robot navigation foundation model or to build one via an ingenious combination of vision and language models. The difference to such approaches is stated in our Related Works in section 6 under “Contrasting recent approaches with our goal”.

Indeed, our goal is stated in our Preliminaries, section 2, part of which is reproduced here for the reviewer’s convenience:

Problem statement: We seek to establish the bare design criteria for training robust, text-instructed, end-to-end control agents. Specifically, our goal is to delineate the conditions for effective leveraging of off-the-shelf models to extract meaningful features suitable for compact downstream policy networks.

In other words, what learning representations are required for the emergence of object and environment generalization in end-to-end navigation? This is a very different research question from that of designing a comprehensive navigation model. Furthermore, in terms of dataset design, we try to answer the question (also in section 2):

Dataset design: What is the minimal degree of data diversity required to obtain sufficiently rich feature representations?

Weakness 3

The reviewer highlights a significant systems limitation we clearly state in our Discussion, Appendix A.2:

Limitations and Future Work: The framework presented in this manuscript is limited to instantaneous decisions. Indeed, the policy can only act with information from the current image and has no access to a history of representations or actions. We are keen to incorporate our potent multi-modal feature encoding scheme into sequential decision-making processes. This would enable Flex to go beyond generalization between environments and objects, and handle instructions over actions, sequences of steps, and behavior modes.

As mentioned, this is left for future manuscripts, as we believe it to be a crucial capability that will enable a range of behavior required for the system to be useful for long-horizon applications.

However, we maintain that this does not dilute the contribution of the paper, which we see as less of a novel systems design but rather an investigation of the relationship between learning representations and the emergence of robust generalization (albeit on a simple scenario).

Indeed, we aim to devise a minimal approach for adapting the generalization capabilities of VLMs to downstream robotics tasks, with minimal training/data/model modification.

We sincerely thank the reviewer for the thorough assessment and constructive feedback, which helped us to enhance our paper's clarity. We are confident that we have adequately addressed all of your concerns. If you have any further questions, we are eager to continue the discussion during the review period. Thank you!

Reply to question 1

We thank the reviewer for touching a crucial point of the work. We remind the reviewer that the focus of this work is the identification of minimal requirements to obtain generalization in navigation to new objects in new scenes (in simulation or in the real world). The natural assumption made is that foundation models are essential in bridging the generalization gap, especially for the handling of language-provided concepts outside the scope of the policy training dataset. With that said, the question becomes about the suitable usage of VLM models (here BLIP-2) to achieve this objective.

Hence, the short answer to the first question is that it does not work. In fact, if we understand the reviewer correctly, the setup described corresponds to the case presented in Figure 3 with patch resolution 1. Here the entire image goes through the VLM which does use its underlying ViT 16x16 patch attention mechanism to encode the image. With an attention based policy trained on such features, the closed loop navigation performance is close to 0% on all tests whether in or out of distribution. The introduction of spatial masking is accompanied by first the emergence of success, and further the improvement of success as the resolution increases (again in Figure 3).

Now to address the point regarding the MLP average pooling, we bring the reviewer’s attention to the fact that the MLP acts on spatially encoded patch features and does not interfere with the encoding process. Thus, no loss of spatial information occurs before the decision policy. The linear layers of the MLP pre-pooling operate on the spatially encoded patch features, learning the appropriate relationships and associations between spatial positions and actions. These operations are completed before average pooling, which serves only as a final aggregation step. Thus, the MLP policy still sustains reasonable levels of success in easy scenarios (Figure 4).

In short: (i) The standard ViT model is not designed to map input images directly to policy outputs. As a result, the final descriptor output of a ViT does not capture the necessary information required for learning a generalizable policy for navigation or other downstream task. Additionally, (ii) The patch features produced by intermediate layers of the ViT before the output stage do not share an embedding space with the text data, as, they are not trained to represent patch semantics fused with text information. To this end, for text-based navigation tasks and many other downstream tasks spatial-text-patch fused features are required. To enable text-patch fusion embeddings, our novel approach is required. Finally (iii) The policy network leverages all the available information from our method, including semantics, text-fusion embeddings, and spatial data, to effectively learn from these diverse sources. So the main idea is to provide the policy network (which is treated as the decision maker) with all possible relevant features! And it will learn to make a decision.

Reply to question 2

We appreciate the reviewer's comment. As we think that this is a very important part of the paper, we would like to clarify it to the best we can. Our ablation study focused on three components that cover all layers of the framework:

1- Feature Extractor Model: We investigated the minimal approach to obtaining a feature extractor that provides all necessary features without additional training, thereby allowing good generalization due to its features. We compared our approach to standard Vision Transformer (ViT) features and found that they do not perform effectively without the high-resolution features. Regarding the use of interlayer patch features, this approach cannot generalize to unseen text data because the patch features are not trained to fuse text with image descriptions. In fact, we initially experimented with this method using CLIP patch features but were unable to generalize text commands to objects not seen in the training data.

2- Policy Network Choices: We experimented with various popular architectures for the policy network, including ViT, Convolutional Neural Networks (CNNs), and Multi-Layer Perceptrons (MLPs). Our ablation study shows slight advantages for ViT, as detailed in Section 5.3, and we are happy to test more if any suggestions are provided.

3- Training Data Importance: We strongly believe that data is crucial. Therefore, we focused on determining the sufficient amount of training data needed in terms of objects, text commands, and more.

We want to emphasize that we are more than happy to consider and adopt any relevant suggestions we receive.