DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution

Thank you for your time and constructive feedback. Please see the response below.

W1/L1: Experiments are limited in simulated environments ; show more analysis which can show that it can be effective in the real world

Thanks. We admit that the submitted version of manuscript limits in the simulated CALVIN benchmark. We are actively expanding DeeR's application to real-world robotic tasks by integrating it with the OpenVLA framework, which is tailored for real-world manipulation, rather than RoboFlamingo. However, training on the Open X-Embodiment Dataset [1] is time-intensive, and setting up a real evaluation platform using the Franka robot arm is a significant workload. Due to the tight timeline of this rebuttal, we regret that comprehensive results are not yet available.

Nevertheless, we have initial results on realistic inference times evaluated on an NVIDIA V100 GPU. DeeR achieves a 68.1% reduction in LLM inference time compared to RoboFlamingo++, which aligns with the theoretical 80.7% reduction in FLOPS. This was accomplished without specific optimization for early-exit implementation, and we expect future optimization to further improve DeeR’s operational efficiency.

Further, we wish to highlight that the CALVIN benchmark are developed for capturing many challenges present in real-world settings. It encompasses a wide array of robotic manipulation tasks and features over 400 crowd-sourced natural language instructions, posing significant challenges to SOTA algorithms due to its realistic complexity.

We believe these points collectively demonstrate DeeR's substantial potential for real-world applications.

W2/L2: Setting optimal thresholds for early exits requires careful tuning. Can this framework solve thresholds effectively in challenging low-data or unseen situations?

Thank you for raising this issue. We would like to point out that DeeR determine optimal thresholds using a specialized algorithm rather than relying on manual trial-and-error methods, As detailed in lines 239-260. To validate the effectiveness of threshold-solving framework in real-world-like scenarios, including low-data environments and situations unseen during training, we conducted further experiments. Given a predined GLOPS budget, we solve thresholds directly using a validation set D or a 0.1% subset of the training data ABC (low-data and unseen situation), then assess the model under these thresholds. The results are as follows:

These experiments demonstrate that our computed thresholds can effectively generalize and be capable of handling challenges typical of real-world applications.

W3: The training is conducted using a limited subset of language-annotated data, which might not capture the full diversity of real-world scenarios.

Thanks for your insightful comment. I would like to clarify that our early-exit strategy for efficient robot execution is not dependent on language-annotated data. The essence of our method lies in dynamically adapting the complexity of the robotic policy based on the specific circumstances. This policy could be a MLLM or a purely visual model, independent of language annotations. In this manuscript, we opted for an MLLM policy due to its demonstrated effectiveness in robotic learning such as RT-2.

Q1: using dynamic networks with the early exiting strategy is a common strategy in efficient LLMs? ; show more about it in related work and highlight your contributions and novelty.

Thank you for your comment. To the best of our knowledge, while early exiting is a recognized strategy in computer vision, its application in LLMs is not fully explored, with only a few related works such as LayerSkip [1] and EELLM [2].

In response to your suggestion, we delineate our unique contributions as follows:

1. We first identified MLLM redandency in robotic scenarios and find that 'easier' circumstances, the majority of scenario, can be efficiently handled by smaller models than typical MLLM (Table 1 of our manuscript).
2. To the best of our knowledge, DeeR is first to introduce a novel early-exit strategy specially designed for robotic MLLM policy, addressing significant challenges in applying early-exit to action generation:

• Metric for Early Exiting: DeeR employs an action consistency metric instead of typical softmax confidence or entropy, crucial for action generation where direct softmax outputs are impractical.
• Threshold Optimization: Previous approaches often relied on manually tuning thresholds or solving threshold on datasets. DeeR uses online learning algorithms to dynamically set thresholds during interactions, enhancing early-exit performance (see Table 2, last row).
• Mitigating Training/dynamic Inference Discrepancy in Temporal Integration: DeeR addresses this by introducing a loss function that randomly samples exits at each timestep, enhancing robustness across dynamic inferences (lines 268-281).

We will expand on these points in the related works of our revision.

Q2: Additional complexity of auxiliary loss

We admit that the impact of the auxiliary loss on performance is modest. However, we retain the auxiliary loss due to its little additional training cost and no extra inference cost, improving the performance for almost free.

[1] Elhoushi, Mostafa, et al. "Layer skip: Enabling early exit inference and self-speculative decoding." arXiv preprint arXiv:2404.16710.

[2] Chen, Yanxi, et al. "Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism." arXiv:2312.04916.

Thank you for your time and constructive feedback. Please see the response below.

W1/Q1: How well the approach generalizes to other robotic tasks and environments? ; Are there any limitations when applying DeeR to different tasks?

In principle, DeeR is designed for general robotic tasks with action outputs, and it can be straightforwardly applied to other tasks and environments. Implementation only involves integrating our early-exit strategy into an existing robotic policy network for the task through multi-exit adaptation. Applying DeeR to additional tasks like navigation and dexterous manipulation is to be explored for future research.

One potential limitation when adapting DeeR to diverse tasks involves calculating action consistency metrics for termination. In tasks that combine different types of actions, such as thw whole body movement and arm manipulation, where certain actions may dominate, a simple L2 distance may not accurately reflect consistency. Addressing this issue is a focus for our future work.

W2/Q3: Comparison with other efficient LLM approaches such as model compression and efficient structural design

Thank you for your suggestion. It's important to note that model compression techniques like quantization and pruning, as well as efficient structural designs, are orthogonal to early-exit strategies like DeeR. These approaches enhance efficiency from distinct aspects. For instance, quantization reduces memory requirements by decreasing the precision of each parameter, while early-exit strategies like DeeR by dynamically skipping unnecessary computation in deeper layers. We take quantization as the representative example to show DeeR can integrate with such techniques:

In future, we plan to further enhance the efficiency of our LLM-based robotic systems by integrating DeeR with both pruning and structural optimizations.

Q2: more details on the implementation of the multi-exit architecture and the early-termination criteria

Thank you for your inquiry. Here are more detailed implementations:

• Multi-exit Architecture: In our approach, the layers of the LLM are segmented into multiple exits at uniform intervals—every two layers in our implementation. At any intermediate exit $i$, we employ a max-pooling operator to aggregate information across the token dimension, serving as the output feature for that exit. During inference, all exits share a single action head for action prediction. The action head includes a lightweight 4-layer LSTM and a 3-layer MLP.
• Early-Termination Criteria: Our termination criteria rely on the consistency of action predictions between consecutive exits. Specifically, if the L2 distance between the action predictions at exits $i$ and $i-1$ is small, we terminate at exit $i$. For the first exit, the input feature of that layer is used as the previous feature for comparison.

Feel free to ask for further details on these aspects.

L1: The effectiveness of DeeR may depend on task-specific optimizations, and the approach may require tuning for different robotic tasks and environments.

Thank you for your comment. DeeR is designed for general manipulation tasks with continuous action outputs and was applied across three CALVIN settings without task-specific optimizations. Adapting to significantly different environments or tasks, like with many existing deep learning methods, often presents challenges that may be attributed to data limitations. In foundational models, scaling up the data and its diversity can often mitigate the need for extensive parameter tuning and task-specific optimization. Addressing data limitations within embodied intelligence remains a compelling direction for future research in the community.

L2: The scalability of DeeR to larger and more complex robotic systems is not fully explored in the paper. Further research is needed to assess the scalability of the approach.

Thank you for highlighting the direction of future research. While the CALVIN benchmark is complex and challenging within simulated environments, we acknowledge the necessity to explore the scalability of DeeR in more diverse and realistic robotic systems. Actually, we are actively expanding DeeR to larger and more complex real-world robotic tasks by integrating it with the OpenVLA framework, which is tailored for real-world manipulation and training on the Open X-Embodiment Dataset. Due to the tight timeline of this rebuttal, we regret that comprehensive results are not yet available. But we believe since OpenVLA is already effective as an MLLM policy for real robot manipulation, incorporating our early-exit strategy should be straightforward.

Conclusion

In conclusion, we appreciate the opportunity to address these vital concerns. We hope that our responses sufficiently clarify your question. If there are any additional questions about our research, please do not hesitate to reach out.

Thank you for your time and feedback. Please see the response below.

W1: uniformly sampling is totally different from the way in inference

We recognize that uniform sampling during training does not perfectly replicate dynamic inference conditions. However, we believe this approach is relatively suitable for dynamic exit for several reasons:

• DeeR functions as a dynamic early exiting system where exit probabilities adjust based on the difficulty of the current scenario and computational constraints. These probabilities are inherently unpredictable. Under limited resources, DeeR may early exit in the way like '11121'; conversely, it might be '66666'.
• Uniform sampling exposes the model to a broad spectrum of scenarios, preparing it for any potential exit sequence during dynamic inference and ensuring it can handle all possible combinations and sequences of exits, including extreme cases.

Nevertheless, exploring alternative sampling distributions can be future work.

W1: Concerns About Similar Layer Output Distributions and Impact on LLM Capability; Why not directly use a small policy network, like rt-1

Thank you for your comments. If trained from scratch with uniform sampling across layers, this degradation may occur. However, our approach effectively mitigates this risk. By adopting the Flamingo methodology, we keep LLM parameters fixed and only fine-tune the cross-attention mechanisms, ensuring stable core language knowledge across layers.

OpenFlamingo progressively enhances representation from one layer to the next. Our DeeR model, with its fixed LLM layers and general knowledge, retains this progressive enhancement, ensuring varied contributions from each layer and preventing distribution convergence. We confirmed this by comparing action inference using feature at the 2nd and 12th layers:

This notable improvement in deeper layers also underscores the benefits of using a larger MLLM model over a simpler alternative like RT-1. By utilizing a pretrained MLLM, DeeR leverages extensive knowledge, learning critical representations for robotic tasks, as evidenced in works like RT-2.

W2: Why still need auxiliary action head, does it really matter?; Does the parameters of action head and aux. head are shared?

We acknowledge that the impact of the auxiliary loss on performance is modest. However, we retain the auxiliary loss due to its little additional training cost and no extra inference cost, improving performance for almost free.

Action head and auxiliary head are not shared. The auxiliary head is discarded after training.

W3: Why DeeR can outperform RoboFlamingo ?

Thanks for your question. DeeR builds upon RoboFlamingo++ and maintains consistent architecture and hyperparameters. The key distinction during training is DeeR's use of the loss function $\mathcal{L}^*$ and an auxiliary loss $\mathcal{L}_\text{aux}$ (detailed in section 3.3). To better understand the performance gains, we conducted an ablation study.

It's important to note that variants without $\mathcal{L}^*$ do not support dynamic exit. Therefore, we evaluated all results at the fixed last exit to assess the upper bound performance of the trained model. While these two losses are initially designed to enable effective early inference, we observe that they also marginally enhance overall performance. We conjecture that this improvement may be due to the introduction of explicit supervision at shallow network layers, making the training process more effective, as seen in similar studies [1]. We will make the ablation clearer in the revision.

W4: Comparison with LayerSkip

Our work and LayerSkip both utilize early exit strategies, but DeeR significantly differs in several ways, as outlined in lines 116-121 of our paper:

1. Dynamic vs. Static Early Exit: LayerSkip uses a static approach, exiting at pre-specified layers for all tokens. Conversely, DeeR dynamically determines exit points based on the complexity of the current scenario, employing a novel termination metric based on action consistency and algorithmically set thresholds according to computational budgets.
2. Action Generation vs. Language Generation: While LayerSkip is tailored for language generation, DeeR is designed for action generation, which necessitates integrating temporal information for decision. This integration introduces challenges like training-inference discrepancies, which DeeR addresses by employing a loss function that samples across different exits at each timestep.

W5: True inference time/frequency in a real environment

Thanks. We admit that the submitted version of manuscript limits in the simulated CALVIN benchmark. We are actively expanding DeeR's application to real-world robotic tasks by integrating it with the OpenVLA framework, which is tailored for real-world manipulation, rather than RoboFlamingo. Due to the tight timeline of this rebuttal, comprehensive results are not yet available. But we believe since OpenVLA is already effective as an MLLM policy for real robot manipulation, incorporating our early-exit strategy should be straightforward.

Currently, we have initial results on realistic inference times evaluated on an NVIDIA V100 GPU. DeeR achieves a 68.1% reduction in LLM inference time compared to RoboFlamingo++, which aligns with the theoretical 80.7% reduction in FLOPS.

We believe these points collectively demonstrate DeeR's substantial potential for real-world applications.

[1] Zhao, Hengshuang, et al. "Pyramid scene parsing network." CVPR. 2017.

We really appreciate your prompt response. We wish to clarify our contributions in comparison to Layerskip.

Although Layerskip's Figure 2 intuitively demonstrates that the optimal minimum layer for predicting the current token may vary, developing methods to dynamically determine the best layer to exit during inference remains a challenge. As illustrated in Sections 4.2 and Figures 6, 8, and 10 of Layerskip, it opts to statically select a fixed layer for exiting. Furthermore, the conclusion of Layerskip suggests exploring "dynamic conditions to determine a different exit layer for each token to improve the token acceptance rate of self-speculative decoding" in the future. This highlights that our approach of dynamically exiting is a unique contribution of DeeR, which develops the action consistency metrics, the threshold solving algorithm, and training strategy for enabling dynamic exit.

We hope this clarification help you understand DeeR. Please do not hesitate to contact us if you have any further questions. Thank you once again for your time!

Thanks for your patience!

You've rightly differentiated between text generation, which often produces multiple tokens, and action generation, where the model extracts a single token representation per language instruction and current image $x_t$. However, it's important to note that decision-making/control is temporal, requiring dynamic inference across timesteps and tasks.

For example, consider two robot tasks—one simpler, one more complex—each requiring about 100 action steps. When Skiplayer is applied to Roboflamingo, it generates one token per action step across both tasks, using fixed LLM layers for all 200 time steps, for instance, L=9. Setting a much smaller L, such as L=2, might not provide sufficient capacity for the more complex task. In contrast, with DeeR, the layers is used dynamically according to the task and specific timestep within the task—e.g., 12235775... for the simpler task and 5678996... for the more complex one. This dynamic inference way along timesteps and tasks allows DeeR the potential to further reduce computational costs.

We appreciate this opportunity to address your concerns. We hope our responses have clarified your questions. If you have any further inquiries about our research, please feel free to reach out.

Thank you for your time and thoughtful review. Please see the response below.

W1/W2: the effectiveness of the proposed method is not verified in real-world robot configuration; Reporting the FPS of the model's operation

Thanks. We admit that the submitted version of manuscript limits in the simulated CALVIN benchmark. We are working hard to extend DeeR to real-world robotics tasks. Specifically, we are integrating our early exit paradigm, DeeR, atop the recent framework OpenVLA [3] rather than RoboFlamingo, considering OpenVLA is an open-sourced VLM-based robotic policy originally designed for real-world manipulation. We are currently training it across a set of 25 datasets from the Open X-Embodiment Dataset [1] as OpenVLA and Octo [2] does. This training process is time-intensive, and setting up a real evaluation platform using the Franka robot arm involves a significant workload. Due to the tight timeline of this rebuttal, we regret that we cannot provide comprehensive results at this time.

Although the real scenario experiment is still on the way, we have preliminary results regarding real operational efficiency. We evaluated both RoboFlamingo and DeeR on the same Nvidia V100 GPU. As the below table shows, DeeR achieves a 68.1% reduction in LLM inference time compared to RoboFlamingo++, aligning with the theoretical 80.7% reduction in FLOPS. This assessment was conducted without code optimization of the early-exit implementation. We anticipate that future optimizations will further enhance DeeR’s actual operational efficiency, closely approaching the FLOPS reduction. We will add these results to the revision.

Further, we wish to highlight that the CALVIN benchmark are developed for capturing many challenges present in real-world settings. It encompasses a wide array of robotic manipulation tasks and features over 400 crowd-sourced natural language instructions, posing significant challenges to SOTA algorithms due to its realistic complexity.

We believe these points collectively demonstrate DeeR's substantial potential for real-world applications.

---

W3: Does early-exit weaken the large language model's ability to understand and follow language instruction? conducting an experiment in the instruction-enriched setting

Thank you for your valuable question. In our model, we maintain the LLM parameters fixed and only fine-tune the vision perceiver and vision-language fusion components, following the training paradigm of Flamingo and RoboFlamingo. Therefore, the overall ability of the whole DeeR model to follow language instructions should be comparable to that of RoboFlamingo.

However, a potential concern is that since the early-exit strategy often utilizes fewer LLM layers than the full model, the language handling capacity of a shallower LLM configuration might be inadequate when dealing with complex (enriched) instructions. To address this, we can adjust the computational budget to use deeper layers and make DeeR have similar performance with RoboFlamingo++. As the following results show, while enriched instructions increase the inference computation cost of DeeR, it still reduces the overall computational cost by ~2.7x compared to RoboFlamingo++ without losing performance.

---

Q1: Why does DeeR perform better on Calvin than RoboFlamingo++

Thanks for your question. DeeR builds upon RoboFlamingo++ and maintains consistent architecture and hyperparameters. The key distinction during training is DeeR's use of the loss function $\mathcal{L}^*$, which randomly samples features over possible exits at each timestep to better mimic dynamic inference scenarios, and an auxiliary loss $\mathcal{L}_\text{aux}$ that optimizes features at intermediate exits (detailed in section 3.3). To further analyze the source of performance gain, we conducted an ablation study below.

It's important to note that variants without $\mathcal{L}^*$ do not support dynamic exit. Therefore, we evaluated all results at the fixed last exit to assess the upper bound performance of the trained model. While these two losses are initially designed to enable effective early inference, we observe that they also marginally enhance overall performance, as shown in the table below. We conjecture that this improvement may be due to the introduction of explicit supervision at shallow network layers, making the training process more effective (similar observations have been made in previous studies [4]). We will make the ablation clearer in the revision.

---

Q2: Can "RGB+Proprio" version of DeeR exceed GR-1?

Thanks for your valuable suggestion to enhance DeeR's performance. Since DeeR was initially designed to improve the inference efficiency of MLLM policies, it was built on the MLLM-based framework RoboFlamingo for a fair baseline comparison without proprioceptive inputs. We believe the potential of enhancing DeeR with proprioceptive inputs to potentially outperform GR-1. However, due to the limited scope of this paper and the extensive time required to train the "RGB+Proprio" version, we are happy to explore it in future.

[1] Padalkar, Abhishek, et al. "Open x-embodiment: Robotic learning datasets and rt-x models." arXiv preprint arXiv:2310.08864 (2023).

[2] Team, Octo Model, et al. "Octo: An open-source generalist robot policy." arXiv preprint arXiv:2405.12213 (2024).

[3] Kim, Moo Jin, et al. "OpenVLA: An Open-Source Vision-Language-Action Model." arXiv preprint arXiv:2406.09246 (2024).

[4] Zhao, Hengshuang, et al. "Pyramid scene parsing network." CVPR. 2017.