VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation

Q1: Are there any statistics from real-world experiment results? I think it's important to understand how the impressive results from simulations can transfer to real-world applications.

A1: We agree that incorporating statistics from real-world experiments would enrich the content of the paper. We are currently working on this and will include the relevant results as soon as they are available.

Q2: You use the RAG and the pretrained speech model to obtain the voiceprint, which is then used to retrieve preferences from the database. It seems the voiceprint is derived from the RAG module. You only present the VLA experiment in a customized task, and I wonder about the performance of the VLA combined with speech RAG in this context. I think this need to be given to support your motivation.

A2: As you summarized, our motivation is to enable the processing of raw speech instructions through a unified end-to-end model, aiming to simplify model architecture and reduce the impact of potential cumulative errors. Additionally, the introduction of raw speech enables a more intuitive extraction of voiceprints, allowing the voice RAG to effectively capture individual intents.

To better analyze the role of the RAG module in our method, we can indeed integrate it with the traditional VLA model, as you suggested. The relevant results are presented in the following table. In this setup, the RAG module can also convey individual intentions to the VLA model, thereby improving its performance on the personalization benchmark.

Q3: I think you need some more ablation studies, such as examining the effects of not using the RAG module in the customized task.

A3: Thank you very much for your valuable suggestions. We have included the corresponding experiments, as shown in the table above.

Weakness1: minor reservations about related work, the confounding nature of speech with and without context (and the lack of a text baseline), the exclusive use of TTS outputs and a few baselines

A1: We appreciate your valuable suggestions and are revising the paper to address these issues. We have enhanced the discussion of related works on multimodal LLMs that incorporate speech processing. Moreover, the experiments in which the RAG context is applied and those in which it is not are further clarified and explained. The text-only baselines used in the experiments have been further detailed and supplemented. In particular, we have conducted experiments using actual speech.

Q1: The RAG pipeline appears to be intimately coupled with speech in this paper, but needn’t be. Why wasn’t a baseline with language only inputs performed? What would be the results?

A1: As you pointed out, the RAG module can indeed be used independently of the speech input. Specifically, in the experiments conducted on the CALVIN benchmark, our model was evaluated using either text or speech instructions. In this part of the study, the RAG module is not utilized when the model processes speech input. Only for the personalization benchmark, the RAG module is used together with speech input.

We have added new baselines using language-only inputs (speech input without voice RAG) on the personalization benchmark, as shown in the table. It is evident that removing the RAG module results in a significant decline in the performance of VLAS, highlighting the RAG’s effectiveness in acquiring and utilizing background knowledge.

Q2: All the experiments appear to be with TTS output, which will certainly be biased. Some results need to be with real speech. Do you have any quantitative numbers with real speakers? Even on a subset of any of the data you presented?

A2: We have conducted new experiments to evaluate the performance of the proposed model when handling real speech instructions, both on the regular CALVIN benchmark and our personalization benchmark. These real speech commands were recorded from 10 persons. It can be found that our model continues to demonstrate good generalization to real audio. Given that our model is trained exclusively on a dataset synthesized from TTS, some performance degradation is acceptable when processing real audio.

Q3: The incorporation of RAG into the pipeline makes some parts of the paper confusing. For example, in Section 4.1 it wasn’t clear if these leveraged the context part of the model (e.g., RAG) (e.g., Table 1).

A3: Sorry for your confusion! The experiments presented in Section 4.1 used only speech commands as input, without employing the voice RAG module. We are revising the description of the relevant sections to address the confusion you mentioned.

Q4: Why wasn’t OpenVLA shown in Table 1?

A4: We did not include OpenVLA as a baseline for comparison in Table 1 due to the following considerations:

a. In terms of network structure, the VLA baseline used in our experiment is highly consistent with the OpenVLA model. The only difference is that OpenVLA employs two visual encoders (DinoV2 and SigLip) to extract distinct visual features, which are then concatenated as the final visual input. In contrast, the experimental VLA was implemented following RT-2, the work that first introduced the VLA concept, and therefore uses only a single visual encoder (CLIP). In other words, the experimental VLA baseline can be considered an approximate self-implementation of OpenVLA.

b. In addition, OpenVLA utilizes the extensive open-source Open X-Embodiment dataset for pre-training, along with several additional training techniques to enhance the model’s performance. These training settings could impact the fairness of the comparative analysis between our model and the openVLA baseline.

Thank you for pointing out some related work and other aspects we may have missed. We are revising the relevant content accordingly.

We are evaluating both the pre-trained OpenVLA model and the fine-tuned OpenVLA model on the CALVIN benchmark. For the pre-trained OpenVLA model, we directly utilized the “openvla-7b-finetuned-libero-10” checkpoint, as it has been tailored for the Franka Emika Panda in a simulation environment, closely resembling our experimental setup. For the fine-tuned OpenVLA model, we followed their officially recommended fine-tuning setup. The CALVIN dataset was converted into the unified RLDS format to enable the fine-tuning of OpenVLA, as they suggested.

Since running the full evaluation of above models on the CALVIN benchmark still requires an additional day, we present their performance on a subset of 30 long-horizon tasks before the end of the discussion. The performance comparison on this sub-dataset is sufficient to demonstrate that our own VLA baseline performs significantly better than the OpenVLA models. It can be observed from the table that:

a. The pre-trained OpenVLA model fails to achieve zero-shot generalization on the CALVIN benchmark.

b. Although the fine-tuned OpenVLA model can complete some tasks in the initial stage of the long-horizon sequence, its overall performance remains suboptimal. We hypothesize that the poor performance is due to its limitation of only supporting third-person view images. Whereas, the image input from the end-effector is crucial for the robot to effectively perform tasks such as picking and rotating. From some other studies[1][2]，we can also find that OpenVLA does not perform as expected. Another potential reason for this phenomenon is that OpenVLA is pre-trained in a large action space, encompassing various embodiments, which may differ significantly from the specific action space of the downstream task. This domain gap may also negatively affect its performance[3]. Therefore, our own VLA model is essential and provides a more reliable baseline for the experiments.

Ref:

[1] Li, Qixiu, et al. "CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation." arXiv:2411.19650, 2024.

[2] Liu, Songming, et al. "RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation." arXiv:2410.07864, 2024.

[3] Szot, Andrew, et al. "Grounding Multimodal Large Language Models in Actions." NeurIPS, 2024.

Weakness1: The necessity of incorporating raw speech, given the availability of video and language (transcripts), remains unclear. Many tasks focus on ownership, and contemporary speech recognition models are capable of discerning individual identities through voice characteristics. Integrating recognized user identity and vocal nuances as metadata in text input might achieve similar effects in current vision-language models (VLMs).

A1: While the majority of current multi-modal LLMs or VLA models demonstrate effective performance in scenarios involving images/videos and textual input, we argue that the integration of raw speech modalities remains valuable for several reasons.

a. Speech represents a unique modality of information, fundamentally distinct from text, as it conveys richer non-semantic content. This non-semantic information, including speaker identity, emotion, and intonation, can enhance the capabilities of VLA and MLLM models, enabling them to better understand and perform assigned tasks. An increasing number of studies have begun to focus on integrating raw speech into Multimodal LLMs, such as GPT-4o [1] and VITA [2].

b. Though it is possible to achieve similar results by employing an additional speech processing module to recognize speech content while simultaneously summarizing vocal nuances as meta-text, this approach presents several challenges. First, the modular pipeline increases the system’s complexity and negatively impacts the robot’s execution efficiency. As shown below, the robot policy employing the ASR+VLA approach consistently lags behind the proposed VLAS model in terms of inference speed during the experiments. Second, in the cascaded pipeline, the pre-trained ASR is designed as a generalist for diverse speech inputs, which limits its sensitivity to robot manipulation commands. As a result, this approach can introduce cumulative errors that significantly harm the model performance. This issue can also be observed in the table and the paper.

In conclusion, while modular solutions are effective, developing a unified end-to-end robot policy model for speech processing remains highly valuable. It may offer an alternative approach distinct from mainstream methods, fostering further scientific exploration.

Reference

[1] https://openai.com/index/hello-gpt-4o/

[2] Fu, Chaoyou, et al. "VITA: Towards Open-Source Interactive Omni Multimodal LLM." arXiv preprint arXiv:2408.05211 (2024).

Weakness2: While speech transcripts alone may lack certain nuances, the video input could compensate for these through visual cues like gestures. The paper’s limited treatment of emotion in its framework is another concern, especially as language models like ChatGPT already offer emotion detection from text ("Emotion detection GPT"), which could outperform humans.

A2: As you mentioned, incorporating visual cues such as gestures is a potentially feasible method. However, this approach is neither the most direct nor the most convenient solution, especially for robot manipulation tasks.

a. In current robotic manipulation tasks, vision sensors are typically positioned to monitor the operation space, either mounted on the robotic arm’s end-effector or placed at a third-person viewpoint focusing on the robotic arm. The person issuing commands is usually outside the visual sensory space, partly to ensure human safety. Therefore, compensating for certain nuances through visual information necessitates additional devices and algorithms to track individual behavior, which inevitably increases the system’s cost and complexity.

b. The VLA model is designed to output Cartesian coordinate actions for the robot to execute, placing greater emphasis on fine-grained location information of objects in the currently observed image. Therefore, though many advanced video-text LLMs exist, in the field of VLA, exemplified by RT-2 and OpenVLA, both utilize straightforward image-text LLMs as the foundation to simultaneously ensure the robot’s performance and efficiency. Capturing human intentions through video can be challenging in terms of both performance and speed.

c. Although this paper does not address emotional features, the proposed paradigm is equally applicable to them. All that is required is to integrate a pre-trained emotion recognition module and a RAG module for emotion processing. We chose voiceprint as the focus of our research because it involves a broader range of scenarios compared to emotions. Since we focus on manipulation tasks rather than conversational tasks like ChatGPT, the impact of emotions on robot manipulation is relatively limited.

While modular pipelines may increase processing time, this impact can be mitigated by leveraging cloud-based ASR solutions, such as the OpenAI Whisper API.

We agree that cloud-based ASR solutions can be effectively utilized. However, they also come with their own set of limitations, such as the high cost, network transmission delays, security and privacy risks. In such cases, our method offers an alternative, allowing users to make choices based on their specific needs. Particularly in the domains of security and privacy, transferring speech data to the cloud is a highly sensitive activity. This is likely to face potential risks of counterfeiting and deception. Thus, many cloud service companies are extremely cautious when it comes to handling speech data.

To RE 3:

Research by Albert Mehrabian indicates that communication is 55% nonverbal, 38% vocal (intonation, tone, etc.), and only 7% words.  This highlights the crucial role of non-verbal cues -- such as eye gaze, body language, and facial expressions -- in effective communication. …. Relying solely on vocal input without integrating human observation risks missing the majority of communicative information. Investigating whether visual cues in videos can already offer complementary or even superior information compared to speech is essential to better understand the added value of incorporating speech.

First, please allow us to state once again that most VLA models currently do not use video as input to understand the scene. This is because video processing may impose a substantial computational burden on robot inference, surpassing the performance improvements it offers. A standard VLA model provides an action frequency of 2-4Hz. However, if the video modality with multiple frames is used, the processing speed could decrease significantly, making it unsuitable for robot manipulation. This is why current state-of-the-art VLAs [1][2][3][4][5] rely on image inputs, and the largest public datasets [5] provide only images either from the end-effector (focusing on the objects) or from a third-person view (focusing on the robot arm, not the person).

Second, as you pointed out, according to Albert Mehrabian, communication is 55% nonverbal, 38% vocal (intonation, tone, etc.), and only 7% words. In current studies on robot policies, visual information (accounting for part of the 55%) and words (7%) have been widely utilized[1][2][3][4][5]. However, efforts to utilize vocal information remain very scarce (perhaps 0%), especially the vocal information account for 38% of the communication. Therefore, we believe that it is imperative to explore the potential use of vocal information.

Therefore, introducing identity information from speech into current VLAs and combining it with the image modality may be a better approach than visual-only methods.

[1] Li, Xinghang et al. “Vision-Language Foundation Models as Effective Robot Imitators”, ICLR, 2024.

[2] Brohan, Anthony, et al. “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control”, CoRL, 2023.

[3] Belkhale, Suneel, et al. “RT-H: Action Hierarchies Using Language”, RSS, 2024.

[4] Kim, Moo Jin, et al. “OpenVLA: An Open-Source Vision-Language-Action Model”, arXiv:2406.09246, 2024.

[5] Padalkar, Abhishek, et al. “Open X-Embodiment: Robotic Learning Datasets and RT-X Models”, arXiv:2310.08864, 2023.

Thank you for your detailed response. We would like to address your remaining concerns point by point.

To RE 1:

Unfortunately, the experiments in the paper do not convincingly demonstrate the unique benefits of using raw speech over speech-converted transcripts. … Similarly, for emotion, text-based emotion recognition has shown strong results

First, we would like to claim that our framework is adaptable for emotion processing. The extraction of speaker and emotion embeddings follows a similar approach, where spectral extraction is applied, followed by a network that compresses it into hidden representations. Then, these representations are used with their corresponding task heads. We use identity information because it is the most suitable for the robot manipulation tasks and covers a wide range of applications. Although emotion, intonation, and other factors may be useful for the VLM, their impact on the VLA task is currently limited.

Second, our paper introduces an end-to-end robot policy that maps raw speech directly to robot actions. This has not been explored in existing research. Meanwhile, the unified architecture helps mitigate cumulative errors, as demonstrated in Section 4.1.

Last but most importantly, when raw speech is used as input, the model offers a potential way for non-semantic content (such as identity) to directly influence actions, which is both intuitive and akin to human interactions. Given the current lack of diverse (speech, action) data, we propose Voice RAG as an alternative solution. However, with sufficient (speech, action) data, our model can inherently learn this personalization, even without the RAG module. In this case, the cascading approach requires designing multiple models, each dedicated to addressing a specific aspect of the original speech. This makes the system more complex and diminishes its extensibility.

To provide compelling evidence, it would be helpful to compare against baselines that incorporate cutting-edge speech recognition and emotion recognition methods. For instance, user identity from speech recognition or emotion metadata from text-based analysis could be appended to each transcript and tested for performance improvements.

In fact, similar experiments have already been presented in our paper. As demonstrated in Section 4.2, we implement a VLA + RAG baseline for comparison with our VLAS. The VLA receives the textual transcript, augmented with near-ground-truth identities from the speaker identification model. The results show that this cascading method does not demonstrate superiority, lagging behind our model by a margin of 4.5%. Moreover, on the standard CALVIN benchmark, where speech is used without identity, our VLAS outperforms the VLA + ASR (Whisper), achieving an average length of 3.70 compared to 3.13.

To RE 2:

The claim that ASR is a generalist and limited for robot manipulation commands may not fully align with real-world usage. … While domain-specific jargon may occasionally be used by robotics experts, the overarching goal of AI and robotics research is to design systems that serve the general public.

Sorry for the misunderstanding. What we intend to convey here is not that general ASR is ineffective. Rather, we believe the model could perform better if it were to incorporate domain-specific knowledge. It's kind of like training people to specialize in a particular task. As demonstrated in Section 4.4, our VLAS-base also works as a general speech recognition model and achieves performance comparable to the Whisper model (2.7% vs 2.79%). However, when used for robot manipulation with domain specific data, our model further help mitigate the cumulative error.

In addition, the robustness of LLMs or VLMs (e.g., ChatGPT) to handle noisy text and infer meaning from context suggests that cumulative errors in an ASR + VLA pipeline might have minimal impact on performance. These models often use context to resolve ambiguities and tolerate inaccuracies in input, making them suitable for natural language tasks.

Notably, although VLA is fine-tuned from VLM, it does not perform as robustly as VLM. Roboflamingo [1] conducts an experiment in which the language instruction for VLA is enriched(using different transcripts with similar meanings). The results show a significant decline in performance. Therefore, the subtle recognition errors from ASR may be amplified by the VLA model, resulting in significant cumulative errors, as demonstrated in Section 4.1.

[1] Li, Xinghang et al. “Vision-Language Foundation Models as Effective Robot Imitators”, ICLR, 2024.

Q1: I wonder if there are additional use case that leverage more semantics information from the speech, like the emotion?

A1: Though our paper focuses on the identity information in speech, the overall pipeline can be readily adapted to tasks involving emotions. All we need to do is integrate a pre-trained emotion recognition module along with an RAG module for emotion handling. We focus on voiceprints in our research primarily because they cover a broader range of scenarios compared to emotions in the context of robotic manipulation.

Some potential use cases for emotion: For example, bringing your favorite food to you when you’re in a bad mood. Or perhaps, when responding to positive emotions, robots might perform tasks at a slower pace but with greater precision. On the other hand, when reacting to negative emotions, robots might prioritize speed, completing tasks more quickly at the expense of some accuracy.

Q2: How long historical information can the model handle? If there are multiple users in history and their ownership violates each other, will the model be able to handle this?

A2: For a single round of dialogue, our model can accept an input length of up to 2048 tokens, matching that of the LLaVA v1.5 model. Currently, our model supports processing only one user’s speech command per round. However, for the robot manipulation task, we treat it as a multi-round dialogue in which the observed images and the speech instruction are updated over time steps. When instructions conflict with each other, the robot will always prioritize the latest user and its preferences. We think this is a very good question. In the future, we may address such situations more flexibly by incorporating different priority levels into the RAG module.

Weakness1: Despite the common issue of inference speed in tuning a large VL model as a VLA model, the authors should still provide an analysis on the time required on using the model, especially the bottleneck (is it the audio processing part?) as well as the possible strategies to boost it.

A1: As you pointed out, analyzing inference efficiency is crucial when fine-tuning a VLM into a VLA model for robotic control. Indeed, this paper employs two key optimizations to enhance the inference speed of the models: downsampling the speech spectrogram and implementing an action update strategy with multi-step prediction and execution.

a. Speech spectrogram downsampling is a widely used strategy to accelerate speech signal processing, where adjacent x-frame spectrograms are aggregated into a single-frame feature through a reshaping operation, effectively reducing the time dimension length. In our experiments, we used the x = 5. Since the effectiveness of this approach has been validated in numerous speech recognition and generation tasks, we did not perform additional related experiments.

b. Given that the state of the environment typically does not change significantly over a short period, our work adopts a simple yet effective multi-step prediction and execution policy. Specifically, we set the number of steps for both VLA and VLAS to r=5. As shown in the following table, when r=5 , both the VLA and VLAS models achieve significant speedups while also demonstrating improved performance on the CALVIN benchmark. The frequency is measured using a single NVIDIA A800 GPU.

We also supplemented the results with an analysis of inference speed and performance for different values of r. The table indicates that r=5 achieves an optimal balance between inference efficiency and manipulation performance.

In conclusion, although the introduction of the speech modality reduces the model’s inference speed, we have mitigated this issue using the aforementioned techniques. Other potential solutions to enhance inference efficiency include techniques such as model quantization and speculative sampling. Our future work will involve the development of a Mamba-based VLAS model to further enhance inference speed.

Weakness2: The evaluation on robotic manipulation is not efficient. It should be evaluated whether the policy generalizable to novel scenes (like training on CALVIN ABC and test on D, or in other simulations) or tasks.

A2: To better evaluate our model’s generalization capability to novel scenes, we conducted experiments where the model was trained on dataset ABC and tested on dataset D. It can be observed that, although all the models experienced performance degradation due to the domain gap, our VLAS model achieved performance comparable to RoboFlamingo and VLA on the CALVIN benchmark, while outperforming the other models.

Moreover, we conducted similar experiments on our personalization benchmark. The results demonstrate that our model is capable of handling novel scenes.

Dear Reviewer PCso:

I hope this message finds you well. This is a gentle reminder regarding the review of our manuscript. We deeply appreciate the invaluable comments and feedback provided by reviewers. They are instrumental in enhancing the quality of our research. As per the schedule, the rebuttal phase is drawing to a close. We understand that you have a demanding schedule and a multitude of responsibilities, but we are keen to receive your feedback before the deadline. This will afford us the opportunity to address any questions or concerns you may have raised in a timely manner. We are eager to incorporate your insights to refine our work and would be grateful if you could share your thoughts prior to the rebuttal deadline.

Thank you very much for your hard work and support. Your dedication to the review process is greatly appreciated.

Thanks to all the authors for their feedback, and I decide to raise my grading. Looking forward to GPT-like interaction with robots via voice commands :D