OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction

W1: OTTER .. performs on par with them on the synthetic Libero benchmark...

The Libero benchmark consists of simple tasks with fixed scenes, distractors, and minimal object variation. As a result, all methods can achieve comparable performance by fitting well to the benchmark’s constraints. Since CLIP is predominantly trained on real-world data, its vision encoder is more effective in real settings than in synthetic ones.

In our real-world experiments, OTTER was evaluated on four primitives, unseen objects, and novel distractors. Experiment videos are available at https://ottericml.github.io/, showcasing OTTER's ability to generalize across diverse environments.

W2: Several of OTTER’s design decisions are not ablated, namely: whether to use Xattn vs Xout from the CLIP vision encoder, Nq=4, and context length set to 12.

We have included the ablation experiments on using X_out. Specifically, we trained OTTER using the X_out from CLIP on the DS-PnP and evaluated on the pick and place tasks. The results are shown in the table below:

These results demonstrate that using X_attn​ significantly improves real-world performance, aligning with findings from ClearCLIP and Appendix Figure 6. Leveraging X_attn enhances OTTER’s generalization in real-world settings compared to using X_out. We also conducted ablations in simulation to assess the impact of context length and N_q​ on performance in Libero-object tasks. The results are shown below:

Overall, our ablation studies support the idea that OTTER’s success is not solely dependent on hyperparameter tuning for a specific setting. Instead, key design choices, such as using X_attn play a more significant role in improving generalization to real-world tasks.

Q1: We define primitive as a fundamental, reusable action or skill that can be applied across multiple tasks when combined with different objects. For example, picking up and pouring are two different primitives, Picking up an apple and picking up a banana are two different tasks but the same primitive.

Q2: For an image input of size 224 × 224, the CLIP vision encoder initially generates 257 tokens (256 from 16 × 16 patches plus 1 CLS token). We extract N_q = 4 text-relevant visual features from X_attn of size 15 d, and concatenate them into a single token. d is the latent dimension. The computational complexity of processing these features is approximately O(4 × 15 d). A standard Transformer’s attention mechanism has a complexity of O(T²d), where T is the number of input tokens. For full-resolution image tokens (T = 257), the computational cost is O(257²d). Thus, the ratio of computational cost when using compressed visual tokens compared to processing all 257 tokens is: (O(4 x15d) + O(d))/O(257^2 d) = 0.001. This demonstrates that our approach reduces the computational cost to approximately 0.1% of the full-token processing cost, leading to significant efficiency gains in both training and inference.

Q3: The action horizon length H=8 is the parameter used in the receding horizon control [1] for action smoothness, as in [2]. For a number of predicted actions of 12, only the first 8 numbers are used for temporal ensembles to calculate the action to execute.

[1] Mayne, D. Q., & Michalska, H. (1988, December). Receding horizon control of nonlinear systems.

[2] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion."

Q4: As in A.2, all models except OpenVLA are given a maximum of 30 seconds per trial. Due to OpenVLA's larger size and slower inference speed, it is allotted 60 seconds.

Q5: Octo has 93 million parameters and an inference speed of 33 Hz. It was trained for 300k steps in 14 hours using a TPU v4-128 pod. Fine-tuning Octo on the DS-ALL dataset for 40k steps takes 6 hours using 8 A100 GPUs. OpenVLA, with 7 billion parameters and a much lower inference speed of 0.5 Hz, was trained for 14 days (21,500 A100-hours) using a cluster of 64 A100 GPUs. Fine-tuning OpenVLA on DS-ALL for 15k steps takes 15 hours on 8 A100 GPUs. OTTER, the smallest with 25.5 million parameters and the highest inference speed of 50 Hz, was pre-trained on the OXE dataset for 40k steps in 12 hours using 4 A100 GPUs. Fine-tuning on DS-ALL for 40k steps takes 6 hours on 8 A100 GPUs.

The authors, however, did not compare their approach with more recent baseline methods like Pi-0.

We thank the reviewer for pointing this out. We have added experiments comparing OTTER with Pi-0 as below.

Specifically, we consider the Pi-0 Fast [1] model trained on Droid. Since our experiment setup is the droid setup, we first evaluate the pi-0 performance directly on our setup, denoted as $\pi_0$-Fast-Droid. As shown in the table, while $\pi_0$-Fast-Droid achieves decent performance on the pick and place primitives, it fails on all the other three primitives as the majority of the Droid dataset is the pick and place primitive. Therefore, we also consider fine-tuning $\pi_0$-Fast-Droid on the DS-ALL, denoted as Finetuned $\pi_0$-Fast-Droid. Finetuned $\pi_0$-Fast-Droid achieves non-zero success rate on Drawer and Poking primitives, but still fails on the pouring primitive. There is also a degraded performance on the Pick and Place primitives for Finetuned $\pi_0$-Fast-Droid. This highlights the difficulty of our experiment setup. OTTER-OXE-L can achieve high success rate on all four primitives on the same amount of demonstrations, indicating that using text-aware visual features extracted from a pre-trained VLM can increase the data efficiency and enhance the generalization ability.

[1] Pertsch, Karl, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. "Fast: Efficient action tokenization for vision-language-action models." arXiv preprint arXiv:2501.09747 (2025).

I noticed that the tasks evaluated are predominantly simple pick-and-place scenarios.

We have primitives other than the pick-and-place as shown in Table 2, where we consider 4 primitives, pouring, drawer, poking and pick-and-place. We have provided the videos for the experiments on https://ottericml.github.io/ to show the diversity of the evaluation tasks.

The authors could include a discussion on related VLA works that do not freeze encoders, such as CogACT.

We thank the reviewer for pointing out this reference. We appreciate and agree with the reviewer’s discussion on CogACT and will include this in the related work of the revised draft.

Have the authors observed task-level generalization of their proposed method, and how does it compare with contemporary approaches?

In OTTER, we define primitive as a fundamental, reusable action or skill that can be applied across multiple tasks when combined with different objects. For example, picking up and pouring are two different primitives, picking up an apple and picking up a banana are two different tasks but the same primitive. OTTER has shown strong task-level generalization as shown in Table 1 and 2. We assume the reviewer is referring to primitive-level generalization.

As the reviewer pointed out earlier, keeping a frozen pre-trained Visual-Language Model (VLM) primarily enhances visual generalization. In OTTER, our focus is on leveraging this strength to improve object-level generalization. While OTTER does not explicitly target primitive-level generalization, it can bring certain benefits in enabling it. By reducing the burden of generalizing to unseen objects, the model can allocate more capacity to learning task primitives and compositional behaviors, which enables more efficient adaptation to novel primitives. We agree that exploring primitive-level generalization is a compelling future direction, and we explicitly encourage work in this area.

Q1: "Effectiveness of text-aware visual feature extraction are not well explored."

We agree that explicitly isolating the impact of our proposed text-aware visual feature extraction is important. The ablation study (DFP-OTTER baseline in Tables 1 and 3 and Appendix D, Table 9) was designed to illustrate the benefit of explicitly selecting text-aware visual tokens compared to directly passing visual features to the policy transformer. Specifically, the results demonstrate that selective text-aware feature extraction substantially outperforms this direct-feature-passing baseline on unseen scenarios.

Q2: "Although the improvement in visual features are reasonable, results in Table.1 shows that embodied features have a large effect on overall performance. Thus, it's quite confusing whether the visual features are the key in this kind of tasks."

While embodied features (proprioceptive information such as robot end-effector pose) significantly contribute to overall performance—given that robotic tasks inherently require spatial grounding—the visual features remain fundamentally critical. If the policy is provided with only embodied features without meaningful visual inputs, the robot cannot perceive objects at all, yielding a zero success rate. Furthermore, pi0 also takes the embodied features. As shown in the additional results in response to reviewer Yr2M, OTTER outperforms pi0, indicating the significance of text-aware visual features.

Q4/W1: "Mainstream VLMs, such as LLaVA, usually freeze CLIP encoder during training rather than finetuning. Another baseline using linearly projected frozen CLIP features shall be included."

Thank you for raising this important point. We address this from two angles:

1. To investigate the effectiveness of simply using frozen CLIP features with linear projection, we conducted a specific ablation using the CLS token from the CLIP encoder as the only visual representation (see Appendix D, Table 9). For convenience, we list the relevant results here: | Method | Unseen Tasks | |-----------------------------|--------------| | OTTER (CLS token only DFP) | 6% ± 0.8% | | OTTER (Ours) | 62% ± 4.2% |

These results show that using just the CLS token with a linear projection under the OTTER formulation—similar to mainstream frozen-encoder approaches like LLaVA—is insufficient for achieving generalization in robotic manipulation tasks.

2. Furthermore, unlike mainstream VLM architectures such as LLaVA, prior state-of-the-art VLA models (e.g., OpenVLA) explicitly demonstrate that fine-tuning the visual encoder significantly improves robotic performance. As reported by OpenVLA (Section 3.4 and Table 1), fine-tuning the vision encoder leads to notably higher success rates compared to using a fully frozen vision encoder. This suggests that directly adopting the mainstream frozen-encoder approach from general VLM literature is not optimal for robotic applications, and our explicit text-aware feature extraction provides an effective alternative solution. We will clearly articulate both points in our revised manuscript to thoroughly address this comparison.

W2: "Beyond text-aware visual feature extraction, visual features are further compressed before input into policy network. Details of this step are not well presented and analyzed."

Thank you for pointing this out. We compress visual features using a learnable attention pooling operation to produce a compact representation, as briefly described in Section 3.2. Specifically, the text-aware visual feature extraction step outputs multiple (m) text-aligned visual tokens (empirically, we used the first 15 text-tokens, which sufficiently covers the instructions) each of dimension 768. Directly concatenating these tokens would result in a prohibitively high-dimensional input (15 × 768) for the policy transformer, significantly increasing computational complexity. Therefore, to mitigate this issue and maintain computational efficiency, we introduce a learnable attention pooling layer to aggregate these visual tokens to reduce the dimension before passing them into the policy. This attention pooling layer has N_q = 4 query tokens and a latent dimension of 64. We further concatenated these 4 tokens into 1 token. We will clarify and expand this step in the revised manuscript, explicitly discussing both the motivation behind the dimensionality reduction and the details of the attention pooling mechanism.

Claims and Evidence

Q1: Clarification on "unseen scenarios"

Thank you for raising this point. In our paper, "unseen scenarios" explicitly refer to tasks involving entirely novel objects or novel combinations of objects and spatial configurations not encountered during training. Crucially, our evaluation—both in real-world (Section 4.1, Appendix A.2) and LIBERO simulation benchmarks—requires the model to distinguish the correct target object from visually similar distractors. Our tasks indeed evaluate visual grounding generalization based on language instructions.

Experimental Designs and Analyses

Q1: Regarding the fairness of baseline comparisons (DFP-OTTER vs. OpenVLA)

We appreciate this concern and clarify our intentions:

1. Purpose of DFP-OTTER Baseline: it demonstrates explicitly that directly passing visual/textual features into a vanilla transformer is insufficient, underscoring the necessity of selective text-aware visual feature extraction for generalization.

2. Compatibility with existing VLM-based VLAs: Our paper introduces a novel VLA architecture (OTTER) specifically designed around text-aware visual feature extraction. Adding such features directly into existing VLM-based models like OpenVLA would necessitate substantial re-training of the base VLM (e.g., Prismatic VLM [1]), exceeding our available computational resources. Exploring this integration is indeed a compelling future research direction, which we explicitly encourage.

[1] Karamcheti, Siddharth, et. al. "Prismatic VLMs: Investigating the design space of visually-conditioned language models." ICML 2024.

Q2: Regarding plausibility of Direct Feature Passing (DFP)

We included the Direct Feature Passing (DFP) baseline specifically as a minimal ablation to clearly demonstrate the benefit of selective, text-aware visual feature extraction. Introducing additional intermediate methods like MLPs or Q-formers within the VLM would divert focus from our primary contribution, which is explicitly leveraging pre-trained CLIP for improved generalization.

Q3: Clarification on experiments involving truly unseen objects

Our experimental setup explicitly uses entirely novel objects, as detailed in Appendix A.2, Table 6. These unseen objects differ not only in color but also in geometry and texture. Each evaluation includes explicit distractors, ensuring robust assessment of the model’s visual-language grounding capability. OTTER's superior performance demonstrates effective grounding on genuinely novel objects.

Q4: Availability of supplementary video material

Supplementary videos demonstrating task complexity and model performance are now available at our anonymous website: https://ottericml.github.io/

Q5: CLIP vs. other VLM alignment methods

While existing VLM-based approaches (e.g., OpenVLA, Pi0-Fast) implicitly learn alignments between text and visual tokens through fine-tuning their respective large-scale pre-trained models (such as Prismatic or PaliGemma), our method explicitly formulates text-aware visual token extraction as a separate, parameter-free process leveraging the frozen pre-trained CLIP model. Thus, the comparison requested by the reviewer is already covered by our existing baseline evaluations against state-of-the-art methods such as OpenVLA (Tables 1–3) and Pi0-Fast (added in this rebuttal). These comparisons illustrate the advantage of the explicit, selective token extraction approach compared to implicit token alignment methods embedded in other VLM-based approaches.

Q6: Suitability of LIBERO for simulation evaluation

We selected LIBERO explicitly due to its usage in recent state-of-the-art VLA works (OpenVLA, π0-Fast-Droid), facilitating fair and direct comparisons. LIBERO supports evaluations involving explicit distractors under varied language instructions, thus providing meaningful assessments of visual-language generalization.

Essential References Not Discussed

Q1: Discussion of embodied CoT and CoT-VLA works

We appreciate this suggestion and will explicitly include and contrast these methods in the updated related work section of the revised manuscript.

Q2: Additional comparison with π0 / its extensions

We include additional comparisons and discussions with π0-Fast-Droid in response to reviewer Yr2M, the current state-of-the-art VLA model, leveraging extensive pre-training on proprietary robot data.

Weaknesses

W1: Dependence on CLIP and complexity of real-world scenarios

We agree that evaluating more complex real-world scenarios is valuable. However, current evaluations (please see updated tables on the website) already demonstrate significant challenges for state-of-the-art models (OpenVLA, Octo, π0-Fast-Droid), validating our approach’s robustness. Exploring tasks with more complex interactions remains an important future direction, explicitly mentioned in the revision.