UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion.

---

Q1: I'm curious about the quantitative performance of multi-modal and visual prediction after the pretraining stage, on top of the action performance currently presented in the paper.

ANS: We would like to clarify that our primary focus is on robot decision-making. Due to limited computational resources, our pretraining stage uses the LLaVA instruction-tuning dataset for multi-modal understanding (MMU) training and the Bridge robot dataset for future visual prediction training. Our pretraining process adapts the general-purpose Show-o model to robotic domain-specific tasks, which may come at the cost of reduced performance in general settings, such as VQA, OCR, and text-to-image generation. In our approach, we expect the VLM to learn visual predictions that are consistent with language. Although this approach may lead to the loss of ability on general tasks, it will be more conducive to visual prediction and low-level action learning. Results on both simulation and real-world show that this adaptation significantly benefits robot control tasks.

---

Q2: Another thing I find interesting, but currently missing in the paper is potentially the insights from the pretraining. Do the two objectives conflict with each other? Or is the pretraining simply just working?

ANS: Thank you for your insightful comments! Pretraining models for both multi-modal understanding and generation is becoming increasingly popular. Many works in the generative modeling field have explored mixed-pretraining approaches, such as TransFusion [1] and Show-o [2]. Their original objective functions focus on multi-modal understanding and text-to-image generation.

Inspired by the success of these models, we designed an objective function tailored for embodied agents, combining MMU tasks with future image prediction. Since our model builds upon the Show-o framework, we observed that pretraining proceeds without conflicts, and the training remains stable without collapse.

[1] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model https://arxiv.org/abs/2408.11039

[2]Show-o: One Single Transformer to Unify Multimodal Understanding and Generation https://arxiv.org/abs/2408.12528

---

Thank you again for your time and effort in reviewing our work! We hope this clarification can solve your concerns!

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments.

---

Q1: About model efficiency

ANS: We would like to clarify that UP-VLA operates at almost the same control frequency as previous VLA methods. We did not use autoregressive image generation, we jointly predict future images and actions with a single forward LLM pass during policy execution, resulting in almost similar computational cost to prior VLA models. The table below presents runtime efficiency based on the same backbone (Showo), comparing models with and without image prediction (measured as the reciprocal of the average inference time over 100 iterations on the same hardware):

Additionally, our method employs an action chunking size of 10. We observed that action chunk can further enhances performance and produces smoother trajectories.

---

Q2: When pretraining on an internet video dataset, if robotic videos are used for future prediction, why not pretrain action prediction simultaneously?

ANS: Thank you for your question. We did not include action prediction in pretraining stage based on these two reasons: (1) Different robot datasets have varying action spaces with distinct meanings, requiring additional design effort to standardize them under a unified output format. (2) We believe that robot trajectory videos provide more consistent representations across datasets and already contain valuable dynamics information.

Following your suggestion, we conducted an additional experiment incorporating robot action learning during pretraining while keeping the architecture unchanged (by padding actions to the same dimensions). The results indicate that this modification provides no benefit. We hypothesize that this is because RGB images already contain sufficient prior information, and the differences in action spaces across robots make direct action prediction less effective.

---

Q3: About details: Which internet videos were used for pretraining? Why are human hands present in Figure 5? Why is there no execution video demonstration?

ANS: We use the Bridge dataset for image-prediction pretraining, as mentioned in Section 5.2. The human hands in Figure 5 serve as disturbances during policy rollout. For your convenience, we have included additional execution details in the following anonymous video links: https://sites.google.com/view/upvla-rebuttal.

---

Q4: About SOTA autoregressive-based VLA methods

ANS: Following your suggestions, we conducted experiments on the influential models pi0, openvla, and octo. Since these methods do not have official test results, we utilize the open-source code to fine-tune and test them on the calvin data (with settings consistent with UP-VLA). The results on the ABC-D task are shown in the table below, where an asterisk indicates the results we reproduced using the open-source code.

---

Thank you again for your time and effort in reviewing our work! We hope our clarification can solve all your concerns and shows the improved quality of our paper!

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments.

---

Q1: About stronger baselines.

ANS: Thank you for your suggestions! We conduct more experiment on Pi0, OpenVLA, and Octo model. Since these methods only conduct experiments in real world which is hard to replicated, we utilize their open-source code to fine-tune and test them on the Calvin ABC-D (with settings consistent with UP-VLA). The results on the ABC-D task are shown in the table below, where an asterisk* indicates the results we reproduced using the open-source code.

For the pi0-fast method, it primarily improves training efficiency by preprocessing actions. In terms of performance, it is almost identical to pi0. Therefore, we only present the reproduced results of pi0 here.

---

Q2: I suggest the author to dig some extra benefits of your investigate, otherwise you have to compare with the sota results. For example, the predicted image is pretty good and could enforce some downstream tasks?

ANS: Thanks for your insightful suggestions. We provide discussions from the following aspects:

(1) Performance: As you noted, OpenVLA and Pi0-series models leverage powerful VLMs to enhance semantic understanding, similar to our UP-VLA, which employs Showo, a Unified VLM. However, we observed that these methods are good at semantic understanding but lack action precision. The results in Table [Q1] highlight their limitations on the Calvin benchmark. We notice the primary failure mode of these methods in CALVIN stems from imprecise actions(e.g., the robot arm correctly approaches the target object based on the command but fails to grasp it). In contrast, our approach mitigates these shortcomings by integrating visual prediction into the VLA model, leading to more precise actions and superior performance, as shown in Table 1 and Figure 6 of our paper.

(2) Additional benefits of our proposed unified VLA training paradigm: Previous VLA models did not fully exploit the temporal and visual information encoded in video datasets. Our approach introduces a scalable training paradigm that integrates MMU datasets, video datasets, and robot datasets. This allows the robot to benefit from the language comprehension capabilities of VLMs while also excelling in fine-grained manipulation tasks with complex dynamics. As you mentioned, we indeed found that precise visual prediction can leads to precise actions, which is a key factor in our method's effectiveness. Some of these precise predictions are visualized in Figure 7. As a result, UP-VLA demonstrates strong performance on both the Calvin benchmark and real-world scenarios.

---

Q3: I think the framework in this paper is hard to train.

ANS: We completely agree that training with multiple objectives across multiple datasets can be more challenging than training with a single objective. Fortunately, several works in the generative modeling field have unified understanding and generation within a single model, such as TransFusion, Show-O, and others. These works provide a valuable foundation for our approach. In this paper, we build UP-VLA upon Show-O, and we have included all the source code in the supplementary material. Feel free to check it for more details on the training process!

---

Q4: May have to carefully design the ablation study

ANS: Yes! We have conducted comprehensive ablation studies by systematically removing different components respectively from our original framework to verify the effectiveness of each training objective. As reported in Table 3 of the original paper, eliminating multi-modal understanding pretraining, bridge image prediction pretraining, or image prediction fine-tuning all result in a clear performance drop.

Also, following Reviewer-dmnE, we conduct extra ablation study on multi-steps. Specifically, we change the length of action chunk with and without visual prediction to validate the effectiveness of our proposed visual prediction objective.

---

Thank you again for your time and effort in reviewing our work! We hope our clarification can solve all your concerns, and we are always ready to answer any further questions!

We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments.

---

Q1: predicting videos does not necessarily lead to a comprehensive understanding of physical dynamics

ANS: Whether video models can truly understand physical dynamics remains an open research question, with arguments on both sides [1]. In our experiments, we found that training with a video prediction objective enables UP-VLA models to generate physically consistent future frames. Moreover, we think that better understanding can be reflected in better decision-making and control capabilities. As shown in Table [Q2], our ablation studies demonstrate that video prediction objective consistently improves robot control performance in all settings. We hope this clarification can solve your concern!

[1] https://openai.com/index/video-generation-models-as-world-simulators

---

Q2: Maybe performance improvements are more likely attributed to forcing the model to predict multiple future outcomes (future steps), as shown in VLAS, OpenVLA-OFT

ANS: Thank you for your insightful question! We agree that multi-step prediction can benefit action learning, which we adopted as the default in our experiments. However, beyond multi-step prediction, our primary goal was to show that predicting future images (RGB modality) also improves action learning by facilitating information transfer from video datasets (image modality) to action learning (action modality).

To further illustrate the impact of multi-step action prediction and RGB image prediction, we conducted a detailed ablation study along these two axes. Our results show that increasing the number of predicted action steps and incorporating future image predictions both contribute to performance improvements. We hope this ablation further clarifies the effectiveness of UP-VLA, which integrates image prediction objectives into VLA model.

---

Q3:The authors reproduced UP-VLA-RT-2 and phi-w/o-mmu, but one type is VLA and the other type is Prediction. Therefore, we should first determine whether the performance differences are related to the VLA backbone. To address this, results for both methods on other types should be implemented.

ANS: Thank you for your comments. The Show-o model is fine-tuned from the Phi LLM on multimodal understanding datasets. Since we aimed to fully remove the influence of multi-model understanding tasks, we choose to start from the original Phi model in the phi-w/o-mmu ablation. Following your suggestion, we conducted a comprehensive comparison to better highlight the advantages of UP-VLA across different backbones. These experiments further demonstrate that both video prediction and multimodal understanding contribute to improved action learning.

---

Q4: Compare with Seer, VPP, and RoboVLMs. Both Seer and VPP also focus on Prediction and achieve excellent performance. RoboVLM falls under the VLA category and also performs very well.

ANS: Thank you for your insightful comments! We would like to argue that UP-VLA has significant difference on learning paradigm with these previous works. Seer and VPP leverage internet video datasets to aid robot learning, while RoboVLM utilizes a pretrained VLM. Each of these methods relies on a single type of dataset—either video datasets or multimodal understanding datasets. To the best of our knowledge, UP-VLA is the first model to leverage both image prediction (video) and multimodal understanding (MMU) datasets for embodied decision-making. Furthermore, our ablation studies confirm that both types of data significantly contribute to the final performance.

We also note that Seer, VPP, RoboVLM, and UP-VLA all achieve an average task completion length exceeding 4.0 on the CALVIN benchmark. However, these three works adopt distinct architectures from UP-VLA and are concurrent with UP-VLA. To the best of our knowledge, UP-VLA achieves the best performance on CALVIN ABC benchmark among none-concurrent works.

---

We hope our clarifications address your concerns and demonstrate the improved quality of our paper! Please feel free to reach out with any further questions. Thank you again for your valuable time!