VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models

Thank you for your efforts in reviewing our manuscript and for your insightful questions. We address each of them in turn below:

Q1: The core weakness of this paper lies in its lack of fundamental technical innovation.

A1: Thanks for your question. Although our work is primarily experimental in nature, it introduces a number of valuable innovations in network architecture, training algorithms, and empirical findings that set it apart from purely benchmark‑driven studies. Specifically:

1. Network Architecture: (1) We are the first to employ multiple planning representations concurrently for VLA planning and policy learning; (2) We are the first to conduct controlled experiments in which architectures remain identical while parameter counts vary; (3) We are the first to integrate a KV‑cache mechanism to extract backbone information for planning heads; (4) We are the first to introduce a VAR‑like structure for image‑generation planning in VLA; (5) We are the first to block lower‑level gradient flow in a hierarchical VLA (pi0.5 is a concurrent work).

2. Empirical Results: (1) We are the first to show that training VLA from scratch can outperform large‑scale, pretrained models such as OpenVLA and pi0.5‑fast; (2) We are the first to systematically compare the advantages and disadvantages of hierarchical versus integrated paradigms—evaluating success rates, training time, generalization, and continual‑learning performance; (3) We are the first to present a comprehensive comparison of different planning representations along the same metrics.

From our experimental conclusions, which directly address the goal of revealing previously unknown limitations of current paradigms that could inspire future research, we report the following findings:

1. Image‑foresight planning outperforms visually‑grounded planning.

2. Hierarchical paradigms generalize better than integrated paradigms.

3. Scaling up model size does not necessarily improve performance on current VLA and manipulation tasks.

4. Network architecture and training algorithms remain critical; we have not yet reached the point where scaling alone suffices.

5. Hierarchical models require substantially more training (though not inference) time.

6. Planning‑based VLA underperforms end‑to‑end VLA in continual‑learning scenarios.

---

Q2: The limited real-world validation—only three simple deformable object tasks—raises questions about the generalizability of findings to practical robotic applications.

A2: Thanks for your question. We have added more real-world experiments in Appendix A.8. Please see Fig. 8. We perform the sanity check experiments in these tasks with VLA-OS-A-S and baseline models (Diffusion Policy and OpenVLA). Due to constraints on time and computational resources, we evaluated only the success rate of the ActionOnly model in order to confirm its viability on real‑world tasks. Results are as follows:

The results show that the relative performance of our model versus the baseline in real‑world experiments mirrors that observed in simulation.

---

This paper provides no qualitative analysis of failure cases, leaving key questions unanswered: Why does explicit planning degrade so severely when trained from scratch?

A3: Thank you for your question. Yes, our analysis following Finding 2 was purely speculative. To rigorously test why explicit planning underperforms implicit planning, we conducted an additional experiment in which we retrained the explicit Integrated-VLA model while blocking gradient flow from the action head back into the planning head; we refer to this variant as VLA‑OS‑I‑E‑NEW. This design is ingenious because it isolates the role of both embedding type and gradient feedback:

1. Compared with VLA‑OS‑I-I, the only difference is that the action head receives the planning head’s information.
2. Compared with VLA‑OS‑I‑E, the only difference is the removal of gradient backpropagation from the action head into the planning head.
3. Compared with VLA-OS-‑H, the only difference is whether the action head’s input is the decoded planning representation (in VLA‑OS‑H) or the raw KV embeddings (in VLA‑OS‑I‑E‑NEW).

Due to computational constraints, we evaluated only the success rates of language, visual, and image‑foresight planning representations on LIBERO-LONG. Results are as follows:

The resulting success‑rate ordering is: VLA‑OS‑I‑E < VLA‑OS‑I‑E‑NEW < VLA‑OS‑I‑I $\approx$ VLA‑OS‑H

These results indicate that:

1. The gradient from the action head to the planning head degrades performance.

2. Implicit planning still outperforms explicit planning.

3. Decoded planning representations yield better performance than raw KV embeddings.

Thank you for your efforts in reviewing our manuscript and for your insightful questions. We address each of them in turn below:

Q1: Lack of analysis on the data recipe used for VLA

A: Thanks for your question. Thank you for your question. Indeed, the choice of data “recipe” is pivotal for training VLA models. We did not analyze the training datasets used in prior VLA work, as they vary widely across studies and thus resist direct standardization. To control the “data” variable in this VLA-OS study, we unified our training corpus by selecting six representative datasets that cover diverse modalities (2D vs. 3D, rigid-body vs. deformable, real-world vs. simulated, and gripper vs. dexterous-hand end-effectors), and annotated each with three planning representations that share unified structures. This controlled variable design allowed us to isolate and rigorously evaluate our proposed research questions.

You may want to ask whether incorporating additional, heterogeneous datasets would affect key performance metrics. While this is indeed an intriguing question, computational constraints precluded exhaustive experimentation over all possible dataset combinations. Instead, we chose to examine one of the most critical scenarios: whether augmenting training with data that includes planning annotations but no action labels improves task success rates (see Section 4.3, Finding 6).

---

Q2: The generalization experiments in this paper are not strong enough to support finding 7

A2: Thanks for your question. We want to say that:

1. The 6%~7% success rate generalization results are normal and common in THE COLOSSEUM benchmarks, as shown in some related works [1][2];

2. To prove that our conclusion is statistically significant, we perform the Chi‑square test for homogeneity on our generalization ability results, where it can determine whether the proportion of "success/failure" of "three different objects" is the same (i.e. whether the success rate of each group is consistent). The null hypothesis H0 is that all groups have the same success rate. We use the results of VLA-OS-A, VLA-OS-I, and VLA-OS-H from the generalization results of Table 2(a), and find that p=0.062, which shows that we can reject the null hypothesis based on a 10% threshold. Thus, our results are statistically significant.

[1] Pumacay, Wilbert, et al. "The colosseum: A benchmark for evaluating generalization for robotic manipulation." arXiv preprint arXiv:2402.08191 (2024).

[2] Shridhar, Mohit, Yat Long Lo, and Stephen James. "Generative image as action models." arXiv preprint arXiv:2407.07875 (2024).

---

Q3: Given that the authors have already brought a series of increasing model parameters, it's still necessary to see if model size is a critical factor (especially for generalization).

A3: Thanks for your question. In the appendix, we report experiments on model scalability conducted on LIBERO‑LONG—please refer to Finding 12 on our anonymous website. While other questions related to model size (e.g., the impact of planning representations at different model parameter scales, generalization performance, and continual‑learning ability across models of varying sizes) are certainly of interest, computational constraints preclude us from addressing them within the rebuttal period. Concretely, as shown in Finding 12, holding all else constant, larger models require more training steps: even after 100K training steps on LIBERO‑LONG, the 7B‑parameter VLA model fails to match the performance of the 0.5B model. Executing 100K steps on our 8×A100 cluster consumes approximately 67 GPU hours, and a single generalization study would at least entail training 4 (4 model sizes) × 3 (3 paradigms) × 3 (3 planning representations) = 36 models, which is around 2400 GPU hours (=100 days). To explore how models of different scales perform on these and other questions, we plan to evolve VLA‑OS into a comprehensive open‑source platform in the future and to seek collaborations with other research groups to carry out these experiments. We hope you will show understanding and flexibility in this regard.

---

Q4: One might want to check out answers to the broad questions raised in the last few paragraphs of this paper's introduction to have a quick go-through of the findings. Paper writing can be adjusted accordingly.

A4: Thanks for your suggestion! We will revise our writing style in the camera‑ready version by condensing the conclusions and positioning them at the end of the Introduction.

Thank you for your thorough review of our paper, for your endorsement of our work, and for the insightful questions you have posed. We address each of your points below in turn:

Q1: Besides designing a platform for unifying existing methods, it would be good for the author to integrate the findings and propose a novel architecture out of it.

A1: Thanks for your question. Although our work is primarily experimental in nature, it introduces a number of valuable innovations in network architecture, training algorithms, and empirical findings that set it apart from purely benchmark‑driven studies. Specifically:

1. Network Architecture: (1) We are the first to employ multiple planning representations concurrently for VLA planning and policy learning; (2) We are the first to conduct controlled experiments in which architectures remain identical while parameter counts vary; (3) We are the first to integrate a KV‑cache mechanism to extract backbone information for planning heads; (4) We are the first to introduce a VAR‑like structure for image‑generation planning in VLA; (5) We are the first to block lower‑level gradient flow in a hierarchical VLA (pi0.5 is a concurrent work).

2. Empirical Results: (1) We are the first to show that training VLA from scratch can outperform large‑scale, pretrained models such as OpenVLA and pi0.5‑fast; (2) We are the first to systematically compare the advantages and disadvantages of hierarchical versus integrated paradigms—evaluating success rates, training time, generalization, and continual‑learning performance; (3) We are the first to present a comprehensive comparison of different planning representations along the same metrics.

Moreover, the VLA-OS family itself integrates the best practices from several state-of-the-art works, such as multi-step historical inputs, action-chunking outputs, and the KV-cache mechanism for extracting Transformer embeddings, and no existing VLA architecture is the same as VLA-OS models.

---

Q2: Can the experiment run on larger datasets like open-X embodiment? Do the authors think the experiment results still hold?

A2: Thank you for your suggestion! Indeed, pre‑training on a larger-scale dataset would enable us to investigate many more interesting questions. However, as you have pointed out, training such models requires substantial computational resources and time, which is an investment that is typically beyond the means of a university lab. Concretely, pre‑training a 7B‑parameter end‑to‑end VLA model with OXE consumes approximately 21,500 A100 GPU hours, which on our 8×A100 machine amounts to roughly 112 days, not to mention the hundreds of additional planning‑related ablation studies we intend to run. Consequently, for this NeurIPS submission, we do not plan to undertake large‑scale experiments.

Nevertheless, we aim to build VLA‑OS into a general‑purpose VLA research open-source platform and, upon acceptance of this manuscript, to seek future collaborations with other institutions for follow‑up works such as novel research challenges that large‑scale pre‑training will bring.

Thank you for your efforts in reviewing our paper. We appreciate your insightful summary and the questions you have raised. We address each of your points below:

Q1: Lack of Information about Dataset Building

A1: We thank the reviewer for highlighting the importance of dataset quality. As noted in Appendix B.1, consistency filtering addresses two issues: (1) incorrect planning and (2) inconsistent planning across trajectories of the same task.

For (1), we report the initial reasoning accuracy based on human verification. For example, in LIBERO-LONG (379 samples), the initial reasoning accuracy is 0.71 (269/379), where most Gemini-generated plans are correct, and a high-quality template can be selected automatically via Gemini. In contrast, in harder datasets like FurnitureBench, nearly all initial plans are incorrect due to long-horizon complexity, and the initial reasoning accuracy can be regarded as 0. In such cases, we manually create the planning template to ensure regeneration works as intended.

For (2), although consistency is hard to quantify, our pipeline guarantees that all annotations for the same task are identical after filtering, as Gemini follows a fixed high-quality prompt. The high quality can be verifiable in the released dataset.

---

Q2: Lack of Analysis about the explicit planning v.s. implicit planning experiments A2: Thank you for your question. Yes, our analysis following Finding 2 was purely speculative. To rigorously test why explicit planning underperforms implicit planning, we conducted an additional experiment in which we retrained the explicit Integrated-VLA model while blocking gradient flow from the action head back into the planning head; we refer to this variant as VLA‑OS‑I‑E‑NEW. This design is ingenious because it isolates the role of both embedding type and gradient feedback:

1. Compared with VLA‑OS‑I-I, the only difference is that the action head receives the planning head’s information.
2. Compared with VLA‑OS‑I‑E, the only difference is the removal of gradient backpropagation from the action head into the planning head.
3. Compared with VLA-OS-‑H, the only difference is whether the action head’s input is the decoded planning representation (in VLA‑OS‑H) or the raw KV embeddings (in VLA‑OS‑I‑E‑NEW).

Due to computational constraints, we evaluated only the success rates of language, visual, and image‑foresight planning representations on LIBERO-LONG. Results are as follows:

The resulting success‑rate ordering is: VLA‑OS‑I‑E < VLA‑OS‑I‑E‑NEW < VLA‑OS‑I‑I $\approx$ VLA‑OS‑H

These results indicate that:

1. The gradient from the action head to the planning head degrades performance.

2. Implicit planning still outperforms explicit planning.

3. Decoded planning representations yield better performance than raw KV embeddings.

---

Q3: Comparison with RoboVLMs

A3: Thank you for raising this point. While RoboVLMs also offers a systematic investigation of VLA models (and is duly cited in our Related Work Section), VLA‑OS differs fundamentally in both research objectives and research scope:

1. Difference in architectural emphasis. VLA‑OS employs a single, unified network architecture to study how to leverage VLA—deliberately fixing the backbone to de‑emphasize architectural design. By contrast, RoboVLMs addresses the lower-level question of how to construct VLA networks, systematically evaluating different VLM backbones, input‑history lengths, action‑sequence lengths, and discrete vs. continuous action spaces. Rather than re‑exploring network design, VLA‑OS builds upon those findings, integrating the most effective components (and complementary insights from other work such as $\pi_0$) into the VLA‑OS model family.

2. Difference in research scope. VLA‑OS is centered on task planning and the interplay between planning and policy learning, exploring issues such as planning data construction, representation selection, and comparative analysis of planning paradigms. In contrast, RoboVLMs focuses on purely end‑to‑end (planning‑free) VLA models versus non‑VLA baselines, evaluating primarily whether VLA yields superior performance and generalization (for example, compared to simple diffusion policies).

---

Q4: Comparison with Human-aware vision-and-language navigation: Bridging simulation to reality with dynamic human interactions

A4: Thank you for bringing this outstanding related work to my attention. We will incorporate this paper into our references in the camera-ready version.

This work presents a significant advance in embodied AI by integrating dynamic human activities into traditional VLN benchmarks, thereby narrowing the sim‑to‑real gap through three key innovations: an egocentric 60° action space, a realistic HA3D simulator populated with SMPL‑based human motion models, and the HA‑R2R dataset that enriches navigation instructions with human activity contexts. This work complements the VLA‑OS studies in robotic manipulation by similarly emphasizing the separation of high‑level planning and low‑level execution through unified architectures and large‑scale data, while extending VLA paradigms to the domain of dynamic, human‑populated environments.