Early Fusion Helps Vision Language Action Models Generalize Better

4.3 Cross-attention pooling.

To reduce the number of tokens, we use $N$ learnable queries $q$, and keys $k$ and values $v$ from $f_{vl}$, and compute the output using cross attention $X_{attn}(q, k, v)$. We concatenate the $N$ output tokens to one single token, which is $f'_{vl}$. We revised the paper (L226) to make it clear.

5. All pick-and-place tasks, unfair to OpenVLA and Octo

We collected new human demonstrations, trained and evaluated models for three new primitives: poking an object, pouring from one cup to a container, and opening or closing drawers. For each primitive, we collect around 200 demonstrations. For more details on the tasks for each primitive, please refer to Appendix Section A in the updated manuscript.

We then evaluate the finetuned models on 15 unseen tasks (10 trials per task) for the primitives and report the performance in the following table (also section 4.5):

Notably, both Octo and OpenVLA fail to complete any unseen tasks for the pouring, drawer, and poking tasks, likely due to a relatively small amount of demonstrations for each primitive. EF-VLA pre-trained on OXE (the pretraining dataset for Octo and OpenVLA) achieves a high success rate on all four primitives on the same amount of demonstrations, indicating that using fused vision language features from a pre-trained VLM can increase the data efficiency and enhance the generalization ability.

6. Significant architectural difference between EF-VLA and baselines

• EF-VLA can potentially benefit from a longer context length and action prediction horizon, which could create an unfair comparison with the baselines. To address this, we extended the context history length of Octo to 10 and matched its action prediction horizon to ours. It is important to note that Octo cannot exceed a context length of 10 due to its inherent design constraints (to match the design of its pretraining). OpenVLA has many tokens per timestep, so its context length cannot be extended to 10, so we still use its default context length. We report the performance in the above table in Answer 5.

• Although EF-VLA has fewer parameters compared to Octo and OpenVLA, we both trained them until convergence on the same datasets. To provide further illustration of EF-VLA's scalability, we also enlarged EF-VLA to have 8 blocks with latent dimension 768, which matches Octo. We then evaluate the EF-VLA and Large EF-VLA on 15 unseen tasks (10 trials per task) for the primitives and report the performance in the following table. We can observe that EF-VLA's performance also scales with model size, with a consistent performance advantage against Octo and OpenVLA.

We appreciate the reviewer for taking the time to review our paper and give constructive comments. We emphasize the contribution of our work and clarify some commonly raised questions in our general response. Here we address your individual questions as follows.

1. Many VLAs are not Late-fusion

Please refer to our general response for the definition of late-fusion.

2. Unclear notations

• Lines 175-178 are standard formulations of self-attention machinism, which employs two residual additions at Lines 177 and 178. $X_{attn}$ is the output after the attention operation, $X_{res}$ is the identical to the origibal $X$. We replaced the notation for $X_{res}$ with $X$ in the revised paper.
• Thank you for pointing this out. We changed Line 212 to Eq. (4).
• Eq. (1)-(3) follow a standard implementation of attention machinism. We use the same notation as in ClearCLIP, but it's also a very general formulation as proposed in [1]. We have explained the meaning of Proj, LN, and FFN in Line 179.
• In Line 212, the cosine similarity is computed as $\hat{f}_l\hat{f}_v^T$, where $\hat{f}_l$ and $\hat{f}_v$ are normalized feature vectors as described in Line 208.

[1] Vaswani et al, Attention is all you need.

3.1 "What's the reason for using final LN for text features but post-attention LN for visual features? The choice needs a justification"

• A standard attention block will perform operations like Line 175-178, eq (1)-(3). Using the final output of an attention block is default, which reflects the training data flow. All of our attention modules, except for the ClearCLIP component, adhere to this standard, as the attention modules in other VLA models do (e.g., Octo, OpenVLA). In other words, the final attention outputs are used consistently for our text features, proprioceptive state features, policy transformer outputs, and all our baselines.
• Following the findings of ClearCLIP, we observed that using the final LN cannot give us detailed vision-language alignment representations (as shown in Figure 6), but the post-attention LN of CLIP can offer us clean correspondence between patch and text. We then use this strategy to extract clean and fine-grained vision-language representations.
• We provide both qualitative justification (Figure 6) and quantitative justification (Table 1 and 2) for this choice.

3.2 "Following this, what's the meaning of another L2 normalization descript in L207? Are they referring to the same operation?"

The L2 operation is used to normalize the features to compute the cosine similarity, which is also adopted by the origial CLIP model. As shown in Line 207, they are both L2 normalization.

4.1 Why first per-patch alignment and then compress it again. Can't we directly do the final pooling?

We first perform per-patch alignment to obtain patch-level vision-language representations. However, as illustrated in Figure 6, many patch tokens correspond to background elements or distractors, which introduce redundant or irrelevant information for the policy. The policy must learn to compress these redundant tokens to extract meaningful visual information effectively, in order to generalize to unseen distractors and backgrounds.

Final pooling: Our LF-VLA baseline is directly doing the final pooling over vision and text features to get $f'_v$ and $f'_l$, and performs significantly worse than our method as shown in Table 2.

Direct cross attention without per-patch alignment (ClearCLIP): We also added another early fusion baseline to justify our design choice. Instead of using ClearCLIP to get $f\'\_{vl}$, we can use language tokens to generate queries, and perform cross attention to obtain $f\'\_{vl}$. We call this baseline EF-VLA (xattn). We compare EF-VLA (xattn) with EF-VLA on 7 held-out tasks, and it performs significantly worse than the original EF-VLA. This demonstrates the importance of early per-patch alignment (ClearCLIP) against directly performing various pooling strategies.

4.2 Why a single token can provide dense visual features to the policy learning framework.

We want to convey that the vision-language correspondence is localized at the patch level, which is better described using the word fine-grained rather than dense. We updated the term in the revised paper. As shown in Figure 6, the correspondence is notably clean, sparse, and fine-grained.

Leveraging this fine-grained and sparse correspondence, we can represent an image using $N$ $k$-dimensional tokens ($N=4, k=64$ in the paper). These tokens are then concatenated to form a single $N*k=256$-dim token.

7. The proposed particular method highly depends on CLIP; The technical contribution is minor considering the existing ClearCLIP.

We respectfully disagree with such characterization. While ClearCLIP demonstrates improvements in fine-grained vision-language alignment for open-vocabulary segmentation, this work addresses a fundamentally different domain and problem: leveraging this alignment for downstream Vision-Language-Action (VLA) tasks, particularly in robotics.

More importantly, the objective of this paper is to emphasize that when designing model architecture for vision language action models, we can consider ways to better utilize the existing alignment that is present in CLIP (i.e. select only the language relevant visual features as the input to the policy network), instead of re-aligning the visual representation to a language model.

Thus, the experimental impact is orthogonal to that of ClearCLIP. We provide evaluations demonstrating that our approach significantly outperforms state-of-the-art late fusion methods (e.g., OpenVLA) and other baselines in both simulation and real-world tasks. These results highlight the impact of the architectural changes in the robotics context, which is distinct from the goals of ClearCLIP.

By addressing these novel challenges and achieving state-of-the-art performance in VLA tasks, our work represents a substantial extension of ClearCLIP’s principles into a new and impactful domain. We hope the reviewers can recognize the broader implications and technical contributions of our approach within this context.

8. Other baselines are not evaluated in simulated environments. Only one simulated environment is used.

Both the original Octo and OpenVLA didn't report any simulation results, so we didn't include their simulation performance in our paper. Given they are pre-trained on real-world robotics dataset, and OpenVLA also adopt pre-trained vision encoders, their real-world performance can be more convincing to be compared with. We mainly use simulation environments to conduct ablation studies. We are working on incorporating more simulation benchmarks, and will include the results in the final version of the paper.

1. Replied in the general response.

3. cluttered/complex/natural backgrounds

As mentioned in the paper, all our real-world experiments contain 2 or 4 distractors. As shown in Figure 6 (the above two rows), there are multiple objects in the scene. The two rows below just provide illustrative examples.

4. "..., How are N and k chosen? How do they affect the capability of following language instructions, vision-language patch alignment, and generalizing to unseen tasks? Is the concatenation the only way to do this fusion?"

We already added those details to the revised paper. We didn't explicitly perform parameter search on N and k, but naturally make $N*k$ to be the hidden dimension for the policy transformer, which is set to 512 (768 for our larger model). The vision-language patch alignment is done in the CLIP model, which is independent of N and k. The other capabilities also come from the CLIP alignment, as our xattn and late fusion baselines also adopt the same N and k but fail to generalize. As long as N and k are expressive enough to carry important information (in our experiments, one token per image with N*k=512 ), it should perform reasonably well. For sure, concatenation is not the only way, it's one straightforward implementation of other equivalent choices.

5. Since the paper aims at generalization capability, why didn't directly evaluate it in zero-shot settings?

First, We don't have access to the pre-training physical scenes, e.g. Google RT Dataset or Bridge Dataset. The pre-trained model (on OXE) cannot zero-shot generalize to our physical setup, which is totally unseen during pre-training. OpenVLA and Octo failed to perform well on this setup after finetuning, not to mention their zero-shot performance. Second, we already evaluate in zero-shot settings on our own physical setup, with unseen objects, distractors, and language instructions that are unseen in our training dataset.

5+6. "...How can we know your implementation is correct?", "How can I know your long-horizon extension is correct?"

To address your concerns about the baselines, we extend them by changing some modules or hyper-parameters in the config for OpenVLA and Octo, following their instructions in their official codebase (e.g. Octo finetuning with new action horizon at https://github.com/octo-models/octo/blob/main/examples/02_finetune_new_observation_action.py). In Octo's own paper, they didn't observe benefits after increasing context length (Appendix E, "We did not observe benefits of increasing the history length further on the few tasks we evaluated on, though other tasks may benefit."), which is consistent with our observations.

At this moment, we are unable to figure out how to address your concerns about whether our implementation is correct. We believe these concerns stem more from subjective opinions rather than an objective evaluation, which we feel goes beyond the bounds of the ICLR code of ethics (https://iclr.cc/public/CodeOfEthics). We respectfully request the reviewer to assess our contribution based on scientific evidence instead of personal opinions. Any deviation from this standard would result in an unfair review.

6. What's the result of the original Octo without extension? Need a case that matches OpenVLA.

The paper already provides the results for the original Octo. We extend that upon your request. OpenVLA is a 7B model that requires a long time to train. However, we also don't think it's necessary to match OpenVLA as we already demonstrated the scalability of our method by introducing a larger model that matches Octo, with consistent conclusions. In fact, when OpenVLA compared with Octo, it also didn't match Octo. It should be considered as an advantage that our model has fewer parameters than OpenVLA but can still perform and generalize better.

7. "you can consider changing the title to be more explicitly focused on "alignment"... From the comparison between EF-VLA (xattn) and EF-VLA, the most important contribution seems to be ClearCLIP."

Thank you, we will change the title accordingly.

ClearCLIP is an important tool for extracting fine-grained vision-language alignment information at the patch level. Our contribution is to pick the correct method to obtain aligned vision-language representations, and effectively leverage them to train a better robotic generalist policy. For example, SigCLIP and DINO play important roles in OpenVLA, but they are not the contributions of OpenVLA.

3. black background We have included attention map examples from the OXE dataset in the Appendix Figure 7 where we show how our method works across different setups and backgrounds.

4. N and k are hyperparameters that were chosen empirically. We did not extensively tune them, but adopted a simple choice (with N=4 and k=64) to match the hidden dimension of the transformer, which already yielded strong performance. Note that training a neural network requires dozens of hyper-parameters (or even more), and people only conduct ablations on the key design choices, otherwise the experiment scale is infeasible. Given that N and k are reasonable values (not abnormally small or large), we believe there are not sufficient reasons to conduct this ablation or reject this work because of that, as these hyper-parameters are not the key design choices, not specifically tuned but simply chosen, and not considered as any parts of our contribution.

   The attention pooling layer for compression in the "Vision-Languague Fusion" block is already stated as "learnable" in Line 237 of the latest draft.

5. We emphasize that EF-VLA can perform token reduction effectively due to the aligned vision-language features. Otherwise, it would be very challenging for the VLA to learn the compression operation from scratch, given the results of Octo, OpenVLA, our LF-VLA, and EF-VLA (xattn) baselines. Therefore, we argue that the major contribution of EF-VLA is the novel architecture design, which uses the pre-aligned vision language features to extract the most important patch tokens, and achieves much higher data efficiency on top of that. While obtaining pre-aligned vision-language features has been studied in ClearCLIP for image segmentation, EF-VLA shows a way of adapting it with the effective token reduction in VLAs for robot learning, achieving much better generalization. We want to emphasize that our way of using vision language features from a pre-trained VLM differs from previous work such as RT-1, RT-2, and OpenVLA as discussed in the related work, and this novelty should be acknowledged.

6. We argue that OpenVLA is finedtuned on prismatic, which is pre-trained on large VLM datasets. It's unfair for EF-VLA to match the size of OpenVLA, as EF-VLA is only trained on robotics datasets without access to a large VLM dataset. Octo is also trained only on robotics datasets. Since EFVLA and Octo are both trained on the same amount of robotics data (OXE), we argue matching EF-VLA with the size of Octo is a fair comparison. We re-emphasize that OpenVLA doesn't scale up Octo for comparison as well. The latest $\pi_0$ model from physical intelligence (https://www.physicalintelligence.company/download/pi0.pdf) also uses a small action model with a backbone VLM for tokenization, without scaling itself to compare with OpenVLA. The community's objective is to develop a scalable generalist robot policy that is appropriately sized for the task, though not necessarily as large as OpenVLA or other large models.

7. We changed the paper title to: "EF-VLA: vision language action models with aligned vision language features for better generalization" for clarification.

The first part, "alignment". This proposed technique heavily relies on ClearCLIP. As in response to 5, we acknowledge the alignment relies on ClearCLIP but we argue EFVLA shows a novel way of using pre-aligned vision language features in VLA models, that achieves better generalization performance.

Unfortunately, the token reduction is not well presented and studied. The reduction also does not match the claim of "fine-grained vision-language tokens" in the abstract. See response to 4. By "fine-grained", we meant that the "vision language tokens" can be used for accurate patch-level localization (e.g. finding object-related patches), similar to the definition of "dense" in ClearCLIP. Furthermore, we believe the reduction operation matches the usage of ClearCLIP very well. ClearCLIP can give us a very clear correspondence map, which can be effectively compressed into a few important tokens, as suggested by its name ClearCLIP.

The third part, due to the difficulty of making experiment settings close (data, architecture, and model size), it's also hard to know whether the generalizability is from fine-tuning a smaller model on a smaller dataset. All three models, EF-VLA, Octo and OpenVLA are pre-trained and fine-tuned on the same data. The architecture design of EF-VLA is our major contribution and we do not see the motivation of changing baselines to use our architecture design. As in response to 6, matching model size to OpenVLA is unfair to EF-VLA. Since EF-VLA (large) has the same model size as Octo, we argue the generalizability is not due to the model size. We additionally argue that showing superior performance over 4 different primitives using a relatively small dataset should be an advantage, highlighting the data efficiency of EF-VLA. As we already show the data scaling capability of EF-VLA, we don't see the reason for us scaling data just to make baselines better. We sincerely hope the reviewer can acknowledge that we have made the comparison to the baselines as fair as it can be to the best of our capabilities, and the experiment results in the paper are sufficient to show the superior performance of EF-VLA to the other baselines.

Thank you for your comments on editing the paper. Section 3.1 already introduced how ClearCLIP gives better vision-language alignment. As in response to 5, the reason that token reduction can perform well is pre-aligned vision-language features, and our baselines LF-VLA and EF-VLA(xattn) already demonstrated this. We have verified the significance of our model design through extensive experiments to the best of our capabilities.

We have changed the title to reflect the reposition. We hope the reviewer understands it's impossible for everyone to evaluate their models using the exact setups as baselines in the field of robot learning. For example, papers [1, 2, 3, 4, 5 ,6 and more] are all using different setups and non-comparable model sizes. Asking for such experiments is unfair and largely discourages researchers with limited resources. We sincerely ask the reviewer to acknowledge the extensiveness of our experiments and that the significant performance gap between our methods and the baselines is sufficient to show the advantages of the proposed method.

[1] Brohan et al., RT-1: Robotics Transformer for Real-World Control at Scale

[2] Brohan et al., RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

[3] Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model

[4] Octo Model Team et al., Octo: An Open-Source Generalist Robot Policy

[5] Black et al., $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

[6] Shridhar et al., Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

We appreciate the reviewer for taking the time to review our paper and give constructive comments. We emphasize the contribution of our work and clarify some commonly raised questions in our general response. Here we address your individual questions as follows.

1. “Potentially Misleading Title and Incomplete Exploration of Early Fusion”

In this paper, "early fusion" specifically refers to the alignment of language and vision occurring before the policy transformer in a relatively earlier stage, whereas "late fusion" is happening in the policy transformer. We revised the paper to make it clear. We adopt ClearCLIP in our approach, but our methods can also benefit from other strategies that fuse features at even shallower layers if they are better than ClearCLIP. For applying early fusion to other models e.g. OpenVLA, although we can modify the SIGLIP encoder in a similar manner, re-training the 7B OpenVLA models within the rebuttal period is challenging due to limited computation resources and time.

We emphasize that our contribution is not exploring different early fusion strategies, but rather highlighting the fact that early fused vision-language representations can significantly improve the generalization ability of VLA models. We updated the title to avoid confusion.

2. “Missing Ablations on Key Design Choices”

• The purpose of keeping $f\'\_l$ is because the language can specify action primitives (e.g. pick, place, poke and pour), while $f\'\_{vl}$ and $f\'\_{v}$ are both visual representations, which encode important task-relevant information such as object locations. So we need both $f\'\_l$ and $f\'\_{vl}$ (or $f\'\_{v}$) to interact with correct objects with specified primitives. The architecture can be symmetric with $f\'\_l$ and $f\'\_v$ (our LF-VLA baseline). We demonstrate that language-filtered visual representation $f\'\_{vl}$ is better than $f\'\_{v}$ by comparing EF-VLA with LF-VLA both qualitatively (Figure 6) and quantitatively (Table 1, 2). For EF-VLA, we didn't add $f\'\_{v}$ alongside $f\'\_l$ and $f\'\_{vl}$, as $f\'\_l$ and $f\'\_{vl}$ already contain sufficient information to specify and execute a task.

• Late fusion using classification token: We replace the cross-attention on patch tokens with the CLS token, and report the success rates across 7 held-out real-world manipulation tasks in the following table. LF-VLA with CLS token almost failed all the held-out tasks, even worse than LF-VLA (patch tokens). This may be because the CLS tokens on the image don't carry enough detailed visual information for manipulation.

  Task	EF-VLA	LF-VLA (CLS)
  yellowcube-in-black-bowl	0.6	0.1
  yellow-cube-in-grey-bowl	0.7	0
  blue-bear-in-pink-bowl	0.6	0
  reddish-in-grey-bowl	0.55	0
  black-dog-in-pink-bowl	0.8	0.05
  blue-sponge-in-pot	0.4	0
  apple-in-black-bowl	0.7	0

We appreciate your valuable feedback to help us improve the paper. We agree that our method would be more impactful by incorporating studies across a wider range of models and early-fusion strategies. Given the current status of the paper, we would like to follow your suggestions to reposition our contribution as advancing VLA models rather than sorely emphasizing early fusion. We can change the title to "EF-VLA: vision language action models with aligned vision language features for better generalization" to reflect this reposition.

We also acknowledge your concerns regarding the smaller number of benchmarks used in our methods compared to Octo and OpenVLA. Unfortunately, we do not have access to their zero-shot evaluation setups (e.g., platforms used for the Google RT Dataset and Bridge Dataset). To address this, we conducted both in-distribution and zero-shot evaluations, with five primitives and diverse objects in our own scenarios, which involve a comparable number of tasks. We are open to incorporating more physical setups in future work.

I appreciate the authors’ effort in revising the title to better reflect the problem the paper addresses and for providing additional observations. The new title is an improvement over the previous version. However, I still have significant concerns that I hope the authors can address to strengthen the paper, and here are some of my suggestions:

1. The authors stated that the contribution is not exploring different early fusion strategies, but rather highlighting the fact that early fused vision-language representations can significantly improve the generalization ability of VLA models. To support this claim, I suggest the following improvements:

(a) Apply Early Fusion Strategies Across Multiple Models: Testing early fusion strategies on at least two VLA models (e.g., OpenVLA) would provide stronger evidence of the claim’s general applicability. I understand that retraining models like OpenVLA may be challenging due to time constraints, and I am not requesting retraining of any specific model. However, applying the proposed fusion strategies to any additional VLA models would significantly strengthen the claims, as no such validation is evident in the current version.

(b) Conduct Detailed Ablation Studies on Early Fusion Designs: It is crucial to evaluate the current design choices comprehensively. For instance, as mentioned in my original review, 'a deeper fusion study that examines the benefits and trade-offs of fusing features at even shallower layers of the vision encoder.' would be valuable. Evidence from works like LLaVA [1] suggests that the final layer of the CLIP vision encoder might not always be the optimal choice; alternatives like the second last layer could yield better results. Such studies would help identify the best practices for applying early fusion strategies.

(c) Evaluate on Other CLIP-like Models: It would be helpful to assess whether similar properties exist in other CLIP-like models, such as SigLIP [2]. This would provide further validation of the paper’s claims.

Without such exploration, the work risks being perceived as a case study rather than a substantial research contribution, limiting its broader impact on the community.

2. Alternative Paper Formulation: Another possible approach would be to benchmark the proposed method against existing generalist VLA models and demonstrate its superior performance under fair comparisons. However, the current paper uses fewer benchmarks than those used in Octo or OpenVLA, making it difficult to justify the general advantage of this design. Pursuing this direction could provide an alternative way to structure the paper and present its contributions.

In conclusion, while the revised title is an improvement, the current version of the paper does not sufficiently support its claims. I recommend further refinement to strengthen its contributions and maximize its benefit to the community. As it stands, I don't think the current version of the paper is ready for the community and I cannot recommend the paper for acceptance.

Reference

[1] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

[2] Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

We appreciate the reviewer for the time to review our paper and give constructive comments. We emphasize the contribution of our work and clarify some commonly raised questions in our general response. Here we address your individual questions as follows.

1. “Confusing Figure 3”

Thank you for the suggestions to improve the clarity of Figure 3. Here the key, query, value should be $\hat{f}_l$, $\hat{f}_v$, and $\hat{f}_v$, respectively. They are fused through a CLIP-like similarity score, but not standard attention mechanism. We changed the figure accordingly.

2. "Choice of CLIP Over Other VLMs, e.g. ALIGN, BEiT, CoCa"

We selected CLIP for its wide usage and community support. In contrast, ALIGN is not open-sourced, BEiT is limited to image-only functionality and cannot provide vision-language representations, and CoCa relies on OpenCLIP, which is trained on a smaller dataset compared to CLIP and offers less flexibility (e.g. for JAX/FLAX users).

Moreover, CLIP has many follow-up works like ClearCLIP, which we adopted as an effective strategy to extract detailed vision-language representations. This approach may not be generally applicable, as the properties of other VLMs remain largely underexplored.

As discussed in the general response, our contribution is not exporing different early fusion strategies, but rather highlighting the fact that early fused vision-language representations can significantly improve the generalization ability of VLA models.

3. "What is the learnable "cross-attention pooling" in line 225?"

To reduce the number of tokens, we use $N$ learnable queries $q$, and keys $k$ and values $v$ from $f_{vl}$, and compute the output using cross attention $X_{attn}(q, k, v)$. We concatenate the $N$ output tokens to one single token, which is $f'_{vl}$. We revised the paper (L226) to make it clear.

Thanks for your clarification and revised paper!

1. It looks like this work focuses on leveraging alignment in VLM. Thanks for the explanation. I think this term is much clearer than early fusion. I would suggest changing the title and the paper to reflect it.

2. The arguments are not convincing.

   • Is the value of early fusion only in the low-data regime? Considering the development of the field (scaling of the data and model for example), will early fusion benefit in the long term? Most of your experimental settings are training from scratch or fine-tuning on a small dataset. It's good to show how a large dataset can also benefit from the approach. In Table 3 (revised paper) you have EFVLA trained on the OXE dataset, but it is still fine-tuned on the real-world demonstrations.

   • I would like to discuss whether CLIP creates "well-aligned" vision-language representations. This connects to the third point that how the proposed approach could generalize to different VLMs. Otherwise, the work is studying properties of CLIP which has a limited scope. More VLMs should be tested.

3. See above. I understand the limited time frame but apple-to-apple comparisons are the key to justify the contributions.

4. I'm confused by the scale of the dataset. Are 200 demonstrations data-efficient for each "primitive"? A missing key baseline is how well a straightforward behavior cloning model (e.g., using DIffusion Policy) can perform under the same amount of training data.