Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization

Dear Reviewer:

We greatly appreciate your recognition of our novel AGNOSTOS benchmark for evaluating cross-task zero-shot generalization and the strong performance of X-ICM compared to mainstream VLA methods. We believe that our clarifications below can address all your concerns and provide you with new perspectives to re-assess our paper’s contributions.

Q.1 Clarification on the construction of AGNOSTOS.

• We clarify the significant contributions of our AGNOSTOS benchmark to the VLA community:
  • Pioneering Cross-task Zero-shot Benchmark: AGNOSTOS is the first systematic benchmark designed for cross-task zero-shot manipulation, addressing a critical gap in the literature where prior work has largely overlooked this challenging setting.
  • Revealing Generalization Limitations: Through extensive evaluation of multiple VLA methods on AGNOSTOS, we found that existing methods achieve success rates below 20% on 23 unseen tasks (Table 2). This insight highlights the limitations of current approaches and provides a standardized framework to guide future VLA generalization research.
• The decision to build upon RLBench was deliberate: RLBench is a widely adopted, high-fidelity manipulation environment used by influential works (e.g., PerAct [27], RVT2 [33]) and generalization benchmarks (e.g., COLOSSEUM [24], GemBench [25]). Leveraging RLBench ensures informative and reliable comparisons with prior methods, maintaining a cohesive ecosystem for the manipulation community rather than introducing a new simulation environment.

Q.2 Transformation between LLM’s input/output and low-level actions.

We provide a detailed description of the transformation process below.

We define the robot’s workspace in the base coordinate system with ranges $X: [-0.3m, 0.7m]$, $Y: [-0.5m, 0.5m]$, and $Z: [0.6m, 1.6m]$. An end-effector action at a given timestamp is represented as a 7-dimensional vector, e.g., $[0.1m, -0.2m, 0.8m, 20°, 90°, 240°, 1]$, where the first three elements denote the 3D position (X, Y, Z), the next three denote the rotation angles around the X-, Y-, and Z-axes, and the last element indicates the gripper state (0 for closed, 1 for open).

We discretize the workspace and angles as follows:

• The 3D workspace is divided into 100 uniform grids along each axis. For a position $p$ in range $[p_{\text{min}}, p_{\text{max}}]$, the discretized value is computed as: $\text{Grid} = \left\lfloor \frac{p - p_{\text{min}}}{p_{\text{max}} - p_{\text{min}}} \times 100 \right\rfloor$
• Each rotation angle is discretized into the range $[0, 72]$ with a $5°$ interval, yielding 72 bins. The discretized angle is: $\text{Bin} = \left\lfloor \frac{\text{angle}}{5} \right\rfloor$
• The gripper state remains unchanged (0 or 1).

For the example action $[0.1m, -0.2m, 0.8m, 20°, 90°, 240°, 1]$, we convert it into LLM's input format:

• X-position: $\frac{0.1 - (-0.3)}{0.7 - (-0.3)} \times 100 = 40$
• Y-position: $\frac{-0.2 - (-0.5)}{0.5 - (-0.5)} \times 100 = 30$
• Z-position: $\frac{0.8 - 0.6}{1.6 - 0.6} \times 100 = 20$
• X-rotation: $\frac{20}{5} = 4$
• Y-rotation: $\frac{90}{5} = 18$
• Z-rotation: $\frac{240}{5} = 48$
• Gripper state: $1$

Thus, the action in LLM format is $[40, 30, 20, 4, 18, 48, 1]$. The inverse process can convert the action in LLM's format into continuous end-effector action.

We will incorporate these details into the revised manuscript to enhance clarity.

Q.3 Clarification on the Voxposer method.

We clarify that since much of the literature (eg, [R3][R4]) in the community classifies Voxposer as a VLA method, we follow this classification. We will give a special explanation of the difference between Voxposer and mainstream VLA in the paper. Thank you for pointing this out.

[R3] A Survey on Vision-Language-Action Models: An Action Tokenization Perspective, 2025.

[R4] Vision-Language-Action Models: Concepts, Progress, Applications and Challenges, 2025

Q.4 Labeling error in Figure 2.

Thank you, we will correct it.

Q.5 Clarification on the spatial perception with discretization constraints.

As detailed in our response to Q.2, we discretize the robot’s workspace into $1 cm^2$ grids (100 grids per axis across X: [-0.3m, 0.7m], Y: [-0.5m, 0.5m], Z: [0.6m, 1.6m]). Given the sparse and scattered object distribution in the RLBench environment, the likelihood of a single $1 cm^2$ grid encompassing multiple object parts is minimal, reducing ambiguity in most cases.

For the "USB" tasks, the success rate is very high, and we found that this is because the detection of "USB" is very accurate. For the "straight rope" task, the detection performance for the "rope head" is poor, which is the main reason for its low success rate. Therefore, we hope that future work can focus on enhancing the 3D spatial understanding ability of models.

Q.6 Failure analysis (sample selection vs. LLM reasoning).

Thank you for this interesting and valuable question. In order to investigate the impact of sample selection and LLM reasoning on model performance, we compared two experimental settings. One is our original X-ICM model, which follows a cross-task zero-shot setting. Another one is that in the testing of unseen tasks, we do not apply the cross-task sample selection, but instead provide a single sample within the unseen tasks, so that we can probe the performance of X-ICM under the one-shot setting.

On each task, by comparing the performance of the model in the cross-task zero-shot setting (both sample selection and LLM reasoning are involved) and the within-task one-shot setting (providing the most favorable within-task sample, LLM reasoning is more involved), we can perform the following analysis and obtain conclusions:

• If the model performs equally in the two situations (eg, within 5% performance difference), it means that LLM's reasoning ability has a greater impact on the task;
• If the within-task one-shot setting is significantly better (eg, more than 5% performance difference) than the cross-task zero-shot setting, it means that sample selection has a greater impact on the task;
• If the cross-task zero-shot setting is significantly better (eg, more than 5% performance difference) than the within-task one-shot setting, it means that LLM's reasoning ability has a greater impact on the task;

In the table below, we compare the model's performance in these two situations. The table follows the same format as Table 2 in our paper (for simplicity, we use "T(task_id)" to represent the original task names in Table 2). From the results, we can see that, for tasks "T3, T7, T9, T10, T12, T13, T15, T17, T19, T20, T22, T23", the sample selection is more important; For the remaining tasks, the LLM reasoning is more important.

Q.7 Clarification on long-horizon task setting.

Thank you for your suggestion to explore X-ICM’s performance on complex, long-horizon tasks in realistic settings. To clarify, our work focuses on the first systematic evaluation of cross-task zero-shot generalization for VLA methods, a challenging and under-explored problem. Learning dynamic semantic generalization across tasks from robot data is inherently difficult, as VLM foundation models lack this capability. Even simple unseen tasks pose significant challenges in this setting, and long-horizon tasks, requiring sequential sub-task execution, amplify these difficulties. We believe these tasks warrant further in-depth study.

At your request, we explore this interesting problem in real-world experiments. We use tasks "put block into bin", "push button", "put bottle into box", "stack blocks", and "pull out the middle block" as seen tasks, where we have 10 demonstrations for each task.

The long-horizon task we evaluated is "clean the table", according the robot's visual observation, we decompose the long-horizon task into three sequential sub-tasks, ie, "stack cups", "put the stacked cups into plate", "place the mango on top of the stacked cups". We evaluated this task over 20 rollouts, with sub-task success contingent on the success of all prior sub-tasks.

For our X-ICM method, the success rates on sub-tasks "stack cups", "put the stacked cups into plate", "place the mango on top of the stacked cups." are 40%, 25%, and 5%, respectively. Thus, the overall success rate for the long-horizon task is 5%.

For comparison, we extend the RoboPrompt (within-task in-context learning) method to our cross-task zero-shot setting by randomly selecting cross-task samples. And the success rates on the three sub-tasks are 10%, 0%, and 0%, respectively. This also demonstrates the effectiveness of our method.

These results align with expectations, as the sequential nature of long-horizon tasks leads to cumulative failure. The cross-task zero-shot setting exacerbates these difficulties. These findings highlight the need for more advanced algorithms to handle long-horizon tasks in zero-shot scenarios.

Dear Reviewer:

We sincerely thank you for recognizing the clarity of our paper and the valuable contribution of our AGNOSTOS benchmark in advancing cross-task zero-shot generalization for vision-language-action models. We believe that our clarifications below can address all your concerns.

Q.1 Comparison to RoboPrompt and KAT

Since Roboprompt and KAT are designed for within-task in-context learning and rely on random demonstration selection, we extend them to our cross-task setting by randomly selecting cross-task samples. To ensure a fair comparison, all methods use the Qwen2.5-72B-Instruct model as the LLM backbone. The results, shown below, demonstrate that X-ICM significantly outperforms KAT and RoboPrompt in both average success rate and stability, owing to our dynamics-guided cross-task sample selection module.

Q.2 Dynamics-guided sample selection Ablations.

We clarify the motivation for our module design and provide additional baseline comparisons to highlight its effectiveness.

(1) Motivation for Dynamics-guided Sample Selection: Our work tackles the challenging problem of cross-task zero-shot manipulation via in-context learning, where selecting relevant cross-task demonstrations is critical. Effective sample selection requires modeling the temporal dynamics inherent in manipulation tasks. Given that only an initial RGB frame and task description are available during testing, we train a diffusion model to predict the final visual observation from the initial observation. This approach enables the model to learn feature representations that encode task temporal dynamics, facilitating robust cross-task sample selection.

(2) Ablation with Additional Baselines: To demonstrate the effectiveness of our dynamics-guided sample selection, we compared it against two general-purpose feature extractors, CLIP and DINOv2, and a random selection baseline. The results, shown below, indicate that CLIP and DINOv2 outperform random selection but are surpassed by our dynamics diffusion model, which achieves higher performance and stability due to its focus on task dynamics.

Q.3 Clarification on the prompt design.

Thank you for your feedback regarding the prompt. To clarify, as our method focuses on cross-task zero-shot generalization rather than prompt engineering, we adapt the prompt structure from RoboPrompt to suit our cross-task setting. We will ensure proper citation of RoboPrompt in lines 225-235 of the revised manuscript to acknowledge this source. Thank you for bringing this to our attention.

Dear Reviewer:

We greatly appreciate your recognition of our sufficient experiments, well-motivated and significant benchmark, and the strong performance. We believe that our clarifications below can address all your concerns and provide you with new perspectives to re-assess our paper’s contributions.

Q.1 Clarification on Dynamics-guided Sample Selection.

(1) Clarification on the motivation:

Our Dynamics-guided Sample Selection module is designed to capture the dynamic semantics critical for cross-task generalization by modeling the transformation from an initial to a final state in manipulation tasks, as detailed in Section 4.2. Unlike standard methods like Video-CLIP, which rely on video sequences and are thus incompatible with our setting—where only a single initial frame is available during testing unseen tasks—our diffusion-based model predicts future observations from this single frame. This enables effective modeling of task temporal dynamics essential for generalization.

To validate the necessity of our approach, we compared it with baselines using CLIP and DINOv2 features for sample selection, alongside a random selection strategy. The results, shown below, demonstrate that our method achieves higher average success rates and lower variance compared to these general-purpose features, as it is trained specifically on manipulation dynamics, unlike the broader datasets used for CLIP and DINOv2.

We will incorporate this analysis and ablation table into the revised manuscript to clarify the motivation and effectiveness of our approach.

(2) Clarification on cross-task generalization:

Regarding the generalizability of the X-ICM framework, we would like to clarify the scope of generalization we address. Due to the "embodiment gap" (e.g., a new robot or environment), virtually all learning-based VLA models require fine-tuning to adapt to a new data domain. Our work therefore focuses on a model's ability to generalize to unseen tasks within the same domain after this initial adaptation. Therefore, we argue that it is acceptable to train our X-ICM model on seen task data.

In addition, for direct transfer to a new benchmark without training, our dynamics diffusion model can be replaced with an off-the-shelf feature extractor like DINOv2. As shown in the ablation in our response to the previous question, this approach outperforms random selection, confirming the inherent effectiveness and transferability of our cross-task sample selection mechanism itself.

Q.2 Comparison between our X-ICM and Roboprompt.

We clarify the key distinctions and novel contributions of our X-ICM method compared to prior work like RoboPrompt: while both leverage in-context learning, they address fundamentally different problems, and our method introduces a crucial, novel component as a result. Specifically,

A New Problem Setting: RoboPrompt focuses on within-task generalization—learning to perform a seen task in a new scenario. Our work is the first to systematically apply and demonstrate the effectiveness of this in-context learning paradigm for the more challenging cross-task zero-shot setting. As shown in our main results (Table 2), this approach significantly outperforms existing VLA models on unseen tasks.

A Critical Selection Strategy: In a within-task setting, randomly selecting demonstrations is a reasonable strategy, as all examples are relevant. However, in the cross-task setting, selecting relevant examples from a diverse pool of different tasks is non-trivial and critical for success. RoboPrompt does not address this challenge. Our core methodological contribution is the Dynamics-guided Sample Selection module, which is specifically designed to solve this problem by identifying dynamically similar demonstrations across different tasks. The following ablation demonstrates that applying a random selection strategy (analogous to RoboPrompt's approach) in our cross-task setting yields poor and unstable results. Our dynamics-guided selection provides a substantial performance gain, highlighting its necessity and novelty.

Q.3 Clarification on the zero-shot definition.

We clarify that our definition of zero-shot learning in the X-ICM framework is both rigorous and consistent with established standards in the literature [R1][R2]. Our 23 unseen tasks are divided into two difficulty levels: Level-1 tasks share partial similarity with seen tasks in either objects or motions (unseen object or unseen motion settings), while Level-2 tasks have no similarity, representing unseen object-motion compositions.

For Level-1 tasks, the partial dissimilarity (unseen objects or motions) aligns with the conventional zero-shot learning paradigm, as supported by the zero-shot human-object interaction (HOI) detection literature, which is closely related to our setting. Specifically:

• CLIP4HOI (NeurIPS 2023) [R1] defines zero-shot HOI detection to include unseen object (UO), unseen verb (UV), and unseen composition (UC) settings, as noted in the caption of Table 1: “Zero-shot HOI detection results on HICO-DET. UC, UO, and UV denote unseen composition, unseen object, and unseen verb settings.” This directly supports our inclusion of unseen object and motion settings in Level-1 tasks.
• KI2HOI [R2] similarly categorizes zero-shot HOI detection into three scenarios: unseen object, unseen action, and unseen combination, as stated in Section II.C (page 2): “Zero-shot HOI detection focuses on transferring knowledge from known HOI concepts to unseen classes. They can be categorized into three scenarios: unseen object, unseen action, and unseen combination.”

[R1] CLIP4HOI: Towards Adapting CLIP for Practical Zero-Shot HOI Detection, NeurIPS 2023.

[R2] Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration, Neural Networks, 2025.

Q.4 Performance without any demonstrations.

We conducted the zero-demonstration experiment and we found that the success rates on all 23 unseen tasks are 0%. This result is expected. Without any in-context examples, the LLM has no reference to follow the action patterns required to complete the manipulation tasks.

Q.5 Clarification on end-to-end in-context learning.

Thank you for suggesting an end-to-end training model like ICRT that supports in-context learning. To clarify, ICRT is designed for within-task in-context learning, where it leverages demonstrations to adapt to new scenarios of a seen task. Extending ICRT to our cross-task zero-shot setting, where models must generalize to entirely unseen tasks, requires significant further investigation, as this problem introduces unique challenges in sample selection and task dynamics transfer, as discussed in Sections 4.2 and 4.3.

Additionally, our results (Table 2) show that strong cross-task performance relies on a powerful VLM backbone (e.g., VLM-based $\pi_0$ performs well while non-VLM-based RVT2 performs badly). However, training an end-to-end VLM-based model for in-context learning, such as adapting ICRT, incurs prohibitive computational costs due to the need for large-scale, diverse training data.

I appreciate the authors’ substantial efforts in the rebuttal and the additional experiments, which addressed many of my concerns. In particular, the clarification regarding the motivation for dynamics-guided sample selection - necessitated by access to only the task’s initial frame - is helpful, as well as is the detailed explanation of the zero-shot definition.

However, a few issues remain after reading the response:

1. Regarding cross-task generalization (Q.1(2)), the authors clarify that their scope focuses on learning-based VLA models, which require fine-tuning for new data domains. However, both Line 107 (“prompt LLMs for action generation”) and Figure 2 involve pre-trained LLMs generating actions in text form, which are not learning-based VLA models. This creates confusion: while the paper identifies limitations in learning-based VLA models, the proposed solution relies on pre-trained LLMs to generate action rather than directly addressing or improving those learning-based VLA models, which appear misaligned with the stated motivation.

2. In addressing cross-task generalization, the authors mention that their dynamics diffusion model can be replaced by an off-the-shelf feature extractor like DINOv2 to enable transfer without training. While dynamics diffusion achieves better performance, it introduces additional training overhead for adaptation to new domains. Although the authors assume initial training on seen tasks is acceptable, this requirement somewhat weakens the contribution.

I would raise my score to borderline, but a little bit towards rejection. If the authors can further clarify these points, I may consider a higher rating.

Dear Reviewer Qo7p,

Thank you for your detailed feedback on our rebuttal. We value your effort and are pleased that our clarifications on dynamics-guided sample selection and zero-shot definition addressed many concerns. Below, we respond to your remaining comments on generalization, clarifying the X-ICM framework’s alignment with our paper’s objectives and addressing potential misunderstandings respectfully.

[Discussion] Issue #1: Clarification on Q1(2) and Misunderstandings.

(1). Misunderstanding of Q1(2):

We believe Issue #1 arises from a misinterpretation of our response to Q1(2). To resolve this, we first provide a more detailed explanation of the original question. The question Q1(2) is: “Specifically, the proposed framework may not be easily transferable to new benchmarks or environments without retraining the diffusion-based image-editing model, thus undermining the claimed generality of X-ICM.”

Detailed Explanation: For the new benchmarks or environments, they are actually new embodiments, so this is actually a cross-embodiment generalization problem (ie, enabling a model to generalize to new embodiments without additional training). However, our work does not focus on cross-embodiment generalization, which is a well-recognized, unresolved challenge in the VLA community.

(2). Scope of Our Work:

The primary focus of our paper is to develop a benchmark for evaluating the cross-task generalization capabilities of various VLA methods. As demonstrated in Table 2, our evaluation encompasses both learning-based VLA methods (e.g., OpenVLA, $\pi_0$) and learning-free VLA methods (eg, VoxPoser). We specifically assess cross-task generalization within a single embodiment.

The reviewer’s comments in Issue #1, e.g., “Regarding cross-task generalization (Q.1(2)), the authors clarify that their scope focuses on learning-based VLA models, which require fine-tuning for new data domains” and “while the paper identifies limitations in learning-based VLA models,” suggest a misunderstanding. We did not intend to express that the scope of our work on the problem of cross-task generalization is limited to learn-based VLA methods. In fact, our evaluation included both learning-based methods (e.g., OpenVLA, Pi0) and learning-free methods (e.g., VoxPoser, RoboPrompt[see response to Q1 of reviewer Xb1Z]). Similarly, the “limitations” mentioned refer to cross-embodiment generalization, not cross-task generalization.

Furthermore, the reviewer’s note on Line 107 and Figure 2 (pre-trained LLMs generating text actions) also needs clarification. In our response, we did not claim that our X-ICM framework is strictly learning-based or learning-free. Our work focuses on systematically evaluating the cross-task generalization of existing VLA methods, regardless of whether they are learning-based or learning-free. The X-ICM framework integrates a learning-based dynamics diffusion module (to improve cross-task sample selection) and a learning-free LLM module (leveraging inherent in-context generalization capabilities). This design is tailored to address cross-task generalization and is evaluated against all existing learning-based or learning-free VLA methods.

[Discussion] Issue #2: Addressing Cross-Task Generalization and Training Assumptions.

(1). Clarification on Cross-Embodiment Transfer:

Regarding the reviewer’s comment in issue #2: “In addressing cross-task generalization, the authors mention that their dynamics diffusion model can be replaced by an off-the-shelf feature extractor like DINOv2 to enable transfer without training,”, this pertains to cross-embodiment generalization. We offer this as a trivial solution for transferring X-ICM to new embodiments without training. More importantly, our work does not claim that X-ICM aims to achieve training-free generalization to arbitrary new embodiments, as this remains an open VLA challenge outside our scope.

(2). Justification for Training on Seen Tasks:

The reviewer’s comment: “Training on seen tasks weakens the contribution,” warrants clarification. We clarify that training the dynamics diffusion module on our cross-task benchmark is reasonable. For our newly developed cross-task benchmark, existing learning-free VLA methods (e.g., VoxPoser) require task/scene-specific adaptations and underperform (Table 2). Learning-based methods achieve better results, but they must be fine-tuned on seen task data (due to the well-known embodiment gap). Training our dynamics module on seen data aligns with these practices and ensures fair comparison.

We hope these clarifications address your concerns. Our work advances VLA research with a robust cross-task benchmark and framework for cross-task generalization, offering valuable contributions. We welcome further feedback and are committed to addressing it promptly.

Thank you again for your detailed feedback and consideration.

Sincerely,

All Authors.

Dear Reviewer:

We sincerely thank you for recognizing the value of our AGNOSTOS benchmark in addressing a gap in current VLA evaluations, its thoughtful task design, and our comprehensive evaluation of pre-trained models. We believe that our clarifications below can address all your concerns.

Q.1 Statistical significance of results.

To address concerns about the statistical significance of our results, we expanded the evaluation to include 6 different seeds for each task, with 50 rollouts per seed, resulting in a total of 300 rollouts per task. Due to the rebuttal word limit, we present updated results for RVT2 (in-domain VLA), R3M-Align (human-video VLA), OpenVLA, $\pi_0$ (foundation VLA), and our X-ICM (7B) across the 23 unseen tasks, as shown below. Task IDs (T1–T23) correspond to those in Table 2 of our paper.

The updated results confirm that the average success rates and standard deviations for Level-1, Level-2, and all tasks remain consistent with our original findings, with no statistically significant differences. Per-task standard deviations are included to provide transparency. High variance in some tasks is attributed to the RLBench environment’s variable scene configurations, a similar phenomenon also observed in Table I of the RVT2 paper [33].

Q.2 More complex real-world tasks.

To address concerns about the simplicity of real-world experiments, we evaluated two additional non-pick-and-place tasks: "open the drawer" and "unscrew the bottle cap." To support these tasks, we expanded the real-world seen task set by including "pull out the middle block," which involves a pulling action similar to "open the drawer." Following the real-world testing protocol described in our paper, we achieved success rates of 20% for "open the drawer" and 0% for "unscrew the bottle cap" across 20 rollouts per task.

These results align with expectations: the pulling action in "open the drawer" resembles that of "pull out the middle block," enabling partial success, whereas "unscrew the bottle cap" involves a unique twisting action absent from our seen task set, leading to failure. If we can expand the scale of real-world data collection (ie, seen tasks are diverse enough), the non-pick-and-place tasks could be solved to a certain extent. Our simulation results (i.e., Table 2) demonstrate that increasing the diversity of seen tasks improves performance on non-pick-and-place tasks.

Q.3 In-domain and oracle cross-domain performance.

To address concerns about evaluating the effectiveness of in-context learning, we provide performance metrics for RVT2, R3M-Align, OpenVLA, $\pi_0$, and our X-ICM (72B) on 18 in-domain (seen) tasks, as shown below, alongside their performance on 23 unseen tasks. These results demonstrate that all methods achieve significantly higher success rates on in-domain tasks after fine-tuning on seen task demonstrations, highlighting the challenge of cross-task zero-shot generalization.

For X-ICM, we use 18 within-task demonstrations to construct in-context prompts for in-domain tasks, yielding a 60.4% average success rate, which is substantially higher than its 30.1% on unseen tasks. However, our X-ICM's performance on in-domain tasks is lower than that of learning-based VLA methods like $\pi_0$ (70.5%). This gap is expected, as X-ICM relies solely on the LLM’s in-context learning capability without fine-tuning.

To estimate the oracle performance of X-ICM on the 23 unseen tasks, we conducted a one-shot experiment, providing a single within-task demonstration per unseen task, selected based on semantic similarity using our dynamics-guided sample selection strategy. This achieved an average success rate of 42.8%, which can serve as an upper limit reference for cross-task zero-shot methods (our zero-shot performance: 30.1%).

Q.4 paper format and grammar correction.

Thank you for the reminder on the paper format. We will move Figure 1 to the end of the abstract section. And we will remove the logo from the title of the paper. We will correct grammar issues based on your useful suggestions.

Q.5 Why did we choose RLBench instead of a new simulation?

We clarify that our contribution extends significantly beyond adding tasks to RLBench.

Our primary contribution lies in introducing a novel evaluation framework focused on cross-task, zero-shot generalization. This framework, supported by a large-scale evaluation of different types of VLA models, provides the first comprehensive analysis of zero-shot generalization capabilities. Our findings (e.g., Table 2) reveal critical limitations in current VLA models and provide the community with a standardized tool to measure future progress.

In addition, our choice of RLBench as the foundation for our benchmark was deliberate. RLBench is one of the most widely used manipulation environments in the community, and it provides a rich ecosystem where many influential methods are already validated. Building a benchmark on this environment allows for immediate, authoritative comparisons against prior work.

Q.6 Finetuning Details.

Thank you for pointing out the need for more detailed descriptions of the fine-tuning process for existing VLA methods. For each VLA method, we follow the official fine-tuning guidelines and explain the details if we use different setups. We will elaborate on all the fine-tuning details in the revised version. Thank you so much for this suggestion.

Q.7 More in-context sample selection baselines.

We compared our dynamics diffusion model with two simpler baselines: CLIP feature-based and DINOv2 feature-based sample selection, alongside a random selection strategy. The table below reports the average success rates and standard deviations for 23 unseen tasks. Both CLIP and DINOv2 baselines outperform random selection in terms of average performance and stability, but they fall short of our dynamics diffusion model. This is because CLIP and DINOv2 can only capture static visual features (only a single initial frame is available during testing each unseen task), which fail to capture the temporal dynamics critical for cross-task generalization. In contrast, our dynamics diffusion model leverages future visual predictions to learn temporal dynamics, which is more effective in sample selection, as detailed in Section 4.2.