RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation

• (Weakness 1) Technical Novelty. Thank you for your detailed comments. We would like to reiterate the technical novelties of RoboMamba in two aspects: 1) an all-round and generalist robotic MLLM framework and 2) a robotic-specific training strategy.

1) An all-round and generalist robotic MLLM framework. The framework designs goal of RoboMamba is to equip the model with high-level reasoning, low-level manipulation actions, and efficiency, all of which are critical attributes in the robotics field. RoboMamba is the first Robotic MLLM framework to effectively balance these three important attributes. Transferring an SSM-based LLM with a completely different architecture into a new task is not trivial. To achieve this, we innovatively chose the Mamba model, which has strong sequence modeling and linear complexity. Conversely, as shown in Q2 2) and Table 3 of the global rebuttal response, simply replacing the LLM in our proposed RoboMamba with another efficient LLMs results in a decline in both high-level reasoning and low-level manipulation capabilities. Meanwhile, as highlighted in global rebuttal Q1, RoboMamba achieves a more efficient reasoning speed than previous robotic MLLMs. The results demonstrate that building a Robotic MLLM with the aforementioned three attributes is a highly meaningful exploration and innovation, requiring appropriate architectural design and a specialized training strategy.

2）Robotic-specific training strategy. We do not simply use a robot-specific dataset (i.e., RoboVQA and manipulation dataset); instead, we meticulously design a training strategy tailored to the robotic domain and our proposed framework. Our training strategy comprises three steps: alignment pre-training, instruction co-training, and robot manipulation fine-tuning. In the instruction co-training stage, we innovatively combine high-level general VQA data with high-level robotic data for integrated training, demonstrating that the fusion of these two data types can mutually enhance each other. As shown in Table 1 and Figure 3 a) of the submitted paper, the results demonstrate that RoboMamba possesses both visual common sense and robot-related reasoning abilities, which previous robotic MLLMs lacked. The performance improvement is due to the robotic data containing a large amount of complex multi-step reasoning data, while the general VQA data includes rich visual scene understanding data. These complement each other, providing RoboMamba with better representation. However, we did not convert low-level manipulation data to high-level data for mixed training. As shown in Table 1 below, we find that the previous SOTA method [12], which uses low-level manipulation data for fine-tuning LLM, leads to a decline in reasoning abilities compared to its pre-trained MLLM model (LLaMA-AdapterV2 [48]). In contrast, for our robot manipulation fine-tuning, we innovatively find that once RoboMamba possesses sufficient reasoning capabilities, it can acquire pose prediction skills with minimal policy head parameter fine-tuning, without compromising the inherent abilities of the MLLM.

• (Weakness 2) Parameter-efficiency. Sorry for the confusion. As indicated in Lines 16, 54, and 71 of the submitted paper, our use of "parameter-efficiency" refers to the minimal parameter fine-tuning (MLP Policy Head) required for learning low-level manipulation skills. It does not suggest that SSM-based LLMs are more parameter-efficient than Transformer-based LLMs.

• (Weakness 3) Real-world experiments. As shown in the bottom right corner of Figure 4 (Pose Prediction), we use a physical Franka Panda robotic arm to manipulate real-world objects. The demonstration video is available in our supplementary material. We apologize for the confusion. We use a red dot to indicate the contact point and a virtual end-effector to show the direction, solely to better demonstrate RoboMamba's predicted 6-DOF pose, as the physical robotic gripper is quite large and occludes the object. Meanwhile, all the reasoning quantitative results in Figures 4 and 5 are based on real-world data from the RoboVQA validation set. Finally, based on your suggestion, we revised the visualization and provided reasoning failure cases in the global rebuttal PDF.

• (Weakness 4) Better representation. Please refer to section 2) Robotic-specific training strategy of Weakness 1.

• (Weakness 5) Ablation study in Table 4. The LLaVA-v1.5 655K mix dataset is already a carefully selected MLLM training dataset [56]. The introduction of additional datasets beyond LLaVA-v1.5 is specifically aimed at enhancing the model's capabilities. We introduce LRV-INSTRUCT 400K to further prevent hallucinations in robotic tasks, which led to improvements in POPE accuracy. Additionally, we included RoboVQA to empower RoboMamba with robot-related reasoning abilities. Finally, as shown in global rebuttal Q2 3), we explored the impact of different training datasets on manipulation abilities.

• (Weakness 6) Scalability. We further explore the scalability of RoboMamba by increasing model parameters and training datasets. The results are shown in Table 1 of the global rebuttal PDF.

• (Questions 1&2). For manipulation experiments, differences in model architecture and training strategies led to varied performance. For reasoning experiments, different MLLM methods [48,52,56,72] are trained on different datasets, as the design of training strategies are core aspects in the MLLM field. For instance, LLaMA-AdapterV2 [48] trains on COCO Caption and ScienceQA; SPHINX [52] uses additional OCR datasets; TinyLLaVA [72] employs a 1246k pre-training dataset from ShareGPT4V. However, these papers directly compare reasoning capabilities across common benchmarks. We will integrate all your valuable suggestions in the final version.

• Weakness 1. Can other architectures work for the proposed framework.

Thank you for the constructive comments. We explore whether other architectures could be applied to the proposed framework from two perspectives: 1) replacing the Image Encoder and 2) replacing the LLM.

1）As you mentioned, we attempt to replace the CLIP encoder used in our submission with a linear computational complexity encoder (XCiT [1]). Meanwhile, we also supplemented the experiment by using SigLIP [c] as an image encoder. As shown in Table 1 below, we explore the impact of different image encoders on reasoning and manipulation abilities. The training dataset and strategy remained consistent across experiments. The results indicate that the choice of image encoder does not significantly affect reasoning ability; rather, the input image resolution is more critical, with higher resolutions improving accuracy. However, using an image encoder without cross-modality alignment (i.e., XCiT) makes it difficult to convert image tokens to LLM language embeddings. Even though our training process includes an alignment pre-training stage, this primarily serves to train the projection layer. Therefore, in future works, we aim to develop a robotic-specific image encoder capable of projecting image tokens into language embeddings while maintaining linear computational complexity. On the other hand, the manipulation results indicate that stronger reasoning abilities in RoboMamba and higher input image resolution aid in policy learning, thereby improving the manipulation accuracy. However, increasing the input resolution also results in additional inference time. Therefore, in our RoboMamba framework, we choose the CLIP (336 x 336) image encoder, which achieves a balance between extracting image semantic knowledge and efficiency.

Table 1. Ablation Study of the Impact of Different Image Encoders

2）We also attempt to replace Mamba with a linear computational complexity LLM in our proposed framework. However, we could not find official published pre-trained parameters for xLSTM [2] in either 2.7B or 1.3B versions. Therefore, we select RWKV-2.7B [d] and Mamba-1.4B, both with linear complexity. As shown in Table 2 below, we explore the impact of different LLMs on reasoning and manipulation abilities. For all experiments, we use the same training data and strategy. The results show that the Mamba-2.7B model possesses superior visual scene reasoning abilities and can more efficiently learn robotic manipulation skills. For other models, due to the sequence modeling capabilities of the LLM itself, the reasoning and manipulation performance achieved is limited. Therefore, we choose Mamba-2.7B as the LLM for our RoboMamba framework to balance high-level reasoning, low-level manipulation actions, and efficiency. Lastly, we will include all the insightful experiments you suggested in our final version. We will also open-source our codebase, continuously experimenting with the latest linear computational complexity architectures to provide a more efficient MLLM framework for the robotics field.

Table 2. Ablation Study of the Impact of Different LLMs

• Weakness 2. Does Mamba has some unique attributes and advantages specially fit the robotic field?

Thank you for the valuable feedback. We would like to explain the robotic-related attributes and advantages of Mamba from the following two perspectives: 1) Balance of Reasoning Ability and Efficiency, and 2) Sequence Modeling Ability.

1）The design goal of the RoboMamba framework is to simultaneously equip the model with high-level reasoning, low-level manipulation actions, and efficiency, all of which are critical attributes in the robotics field. For reasoning abilities, as shown in Table 2 of the Weakness 1 response shows that RoboMamba exhibits stronger reasoning and manipulation capabilities than other linear LLMs (e.g., RWKV, Phi2). For inference efficiency, which is crucial in robotic manipulation and a major challenge for existing MLLM-based policy methods, RoboMamba-2.7B achieves a control frequency of 9.0 Hz. This is significantly faster than other robotic MLLMs, thanks to Mamba's linear computational complexity. Therefore, to balance the attributes needed in the robotics field, we choose Mamba as our LLM, as it possesses both context-aware reasoning ability and efficiency.

2）Meanwhile, the Mamba architecture demonstrates strong sequence modeling ability [22] by generating hidden-space output tokens autoregressively, which effectively captures complex semantic information. During the robot manipulation fine-tuning phase, we only need to fine-tune a simple and efficient MLP policy head to convert Mamba’s output tokens into action poses, rather than using a complex policy head for decoding. This significantly enhances RoboMamba's learning efficiency when facing new manipulation categories or tasks.

• Weakness 3. Figure 1 improvement.

As shown in the global rebuttal PDF, we refined Figure 1 by using arrows with consistent formats and colors, aligning the bounding blocks, and replacing them with a higher-resolution image.

Reference: [c][d] (Global rebuttal);

• (Weakness 1). Comparison of RoboMamba-2.7B with TinyLLaVA

Thank you for the constructive comments. The design goal of RoboMamba is to introduce a new paradigm for adapting Mamba to multimodal robotic tasks, resulting in an innovative Robotic MLLM that integrates high-level reasoning, low-level manipulation actions, and efficiency. Our primary focus is on the robotic domain rather than general multimodal scenarios. We select various general scene MLLM benchmarks to showcase our model's generalizability. However, since RoboMamba and TinyLLaVA use different training datasets, this directly affects their scores on various benchmarks. For example, as shown in Table A1 of TinyLLaVA [81], using ShareGPT4V training datasets with different model architectures can improve MM-Vet performance. Additionally, to more comprehensively compare RoboMamba with TinyLLaVA in the robotic domain, we conducted additional experiments using high-level robotic reasoning benchmarks (RoboVQA) and low-level manipulation pose prediction.

1）For a fair comparison on RoboVQA, we load TinyLLaVA's parameters and fine-tune it on the RoboVQA training set. As shown in Table 1 below, RoboMamba achieves superior performance across BLEU-1 and BLEU-4, indicating that our model possesses advanced robot-related reasoning capabilities. Meanwhile, TinyLLaVA shows stronger reasoning abilities than LLaMA-AdapterV2 (7B), highlighting its significant potential for robotic-related tasks.

2）To compare TinyLLaVA in the manipulation, we use the same fine-tuning strategy by only fine-tuning a simple MLP policy head after the MLLM. As shown in Table 2 below, RoboMamba achieves promising results compared to TinyLLaVA and other methods. These results further support our finding that RoboMamba’s strong reasoning capabilities and framework design enhance the learning of manipulation pose prediction.

On the other hand, inference efficiency is a crucial attribute for robotic manipulation models and poses a major challenge for existing MLLM-based methods. As shown in global rebuttal Q1, RoboMamba-2.7B achieves a control frequency of 9.0 Hz, while TinyLLaVA, under the same inference settings, only achieves 3.9 Hz. These results demonstrate that our approach not only delivers robust performance in the robotic domain but also offers superior efficiency and practicality. Finally, we will incorporate TinyLLaVA's dataset into our co-training stage and update all the aforementioned experiments in the final version.

• (Weakness 2 & Question 2). The analysis of Manipulation results

We provide a comprehensive analysis of our method's manipulation results compared to the previous SOTA method (ManipLLM), especially for delicate objects. We conduct our analysis from two aspects: 1) different test strategies, and 2) more manipulation fine-tuning parameters.

1）During inference, after contacting the object with the predicted 6-DOF pose, following where2act [45], our method simply pulls the objects along the predicted z-axis of the end effector pose. In contrast, ManipLLM adopts an 'Active Impedance Adaptation Policy (AIAP)', which heuristically selects the best pulling direction from all proposed directions (i.e., x-axis, y-axis,and z-axis). This strategy significantly increases the testing time, as it requires trying each predicted direction and then selecting the optimal pulling direction. Therefore, we re-evaluated ManipLLM by testing its accuracy in pulling objects along the z-axis. This adjustment resulted in decreased performance, particularly for delicate categories, as shown in Table 3 below. Consequently, we conclude that the improved performance of ManipLLM in some categories is attributed to the heuristic design of the AIAP interaction rather than to the model's learning capabilities.

2）However, for some delicate categories, RoboMamba still fails to achieve SOTA performance, such as Display, Mouse, Remote, Toilet, and Faucet. This leads us to propose that delicate categories are more challenging to manipulate, so we need to update more parameters during the manipulation learning process. As shown in Table 4 below, increasing the number of MLP layers to 5 improves performance to SOTA levels. Your question is very insightful, and we will continue to explore more detailed experiments.

• (Question 1 and Limitation 1&2). More complex robot manipulation tasks

RoboMamba can support more complex robot manipulation tasks, such as multi-step closed-loop tasks. For the closed-loop benchmark, we selected Meta-World [e], a multi-step tabletop environment. We choose six tasks from Meta-World: Assembly, Bin-Picking, Box Close, Coffee Pull, Hammer, and Button Press. Specifically, we continuously input the current state image to predict the next end-effector pose. Additionally, we utilize an extra MLP to encode the robot's current state and concatenate it with the image features. As shown in Table 1 of Reviewer CsAK, RoboMamba still achieves satisfactory results in the relatively more complex multi-step manipulation experiment. For Limitation 2, instead of heuristically setting the manipulation direction, RoboMamba can also control the direction through closed-loop trajectory prediction. Finally, we will present more closed-loop experiments in the final version.

• (Weakness 3) We will correct all the typos in the final version.

Dear Reviewer iHpb,

We sincerely appreciate your recognition of our work. We will include the closed-loop experiments in the main text and appendix of our paper. May I know if it is possible for you to consider raising your rating above a borderline score if your concerns have been addressed. Thank you very much!

Paper 5916 authors

• (Weakness 1 & Question 1). Additional close-loop experiments

1）Thank you for the constructive comments. RoboMamba is not limited to performing single-step open-loop action predictions. In the submitted paper, for fair comparison, we follow the experimental settings of the latest Robotic MLLM (ManipLLM [12]) to implement action pose prediction for articulated objects. RoboMamba can also efficiently generalize to closed-loop policy learning. Specifically, instead of directly predicting the final contact pose of the end-effector, we continuously input the current state image to predict the next end-effector pose. Additionally, we utilize an extra MLP to encode the robot's current state and concatenate it with the image features. For the close-loop benchmark, we select Meta-World [e] to validate RoboMamba's closed-loop action prediction capabilities. Meta-World is a collection of tasks in which agents command a Sawyer robot arm to manipulate objects in a tabletop environment. We consider six tasks from Meta-World: Assembly, Bin-Picking, Box Close, Coffee Pull, Hammer, and Button Press. As shown in Table 1 below, RoboMamba still achieves satisfactory results in the closed-loop experiment. Finally, we will present more closed-loop experiments in the final version.

Table 1. The closed-loop experiments of RoboMamba, with the success rate metric.

2）Since the Mamba LLM used in RoboMamba balances context-aware reasoning ability with linear computational complexity, inference speed is not a limiting factor for RoboMamba in learning closed-loop policies. We provide detailed inference speed information in global rebuttal Q1.

• (Weakness 1). How much training data is being used and how much variation there is in testing the policy.

The size of the training dataset (10K) is described in Lines 218-223 of the submitted paper. We provide details of the simulator data collection and categories in Lines 603-612 of Appendix B. Regarding the variation between training and testing data, we followed the data collection settings of where2act [45] and ManipLLM [12]. The specific variations can be divided into two aspects: 1) asset variation and 2) state variation.

1) Asset Variation: We use 20 categories from PartNet [42] for seen objects and reserve the remaining 10 categories for unseen objects to analyze if RoboMamba can generalize to noval categories. Specifically, we further divide the seen objects into 1037 training shapes and 489 testing shapes, using only the training shapes to construct the training data. Thus, the shapes of the seen objects encountered during training and testing are different. For unseen categories, there are a total of 274 shapes, which are used exclusively in the testing data.

2) State Variation: We observe the object in the scene from an RGB-D camera with known intrinsics, mounted 4.5-5.5 units away from the object, facing its center. The camera is located at the upper hemisphere of the object with a random azimuth between [-45, 45] and a random altitude between [30, 60]. Since the tasks involve 'pulling,' we also initialize the starting pose for each articulated part randomly between its rest joint state (fully closed) and any position up to half of its joint state (half-opened). These state settings are utilized for both training and testing data, aiming to boost the model's generalization ability.

• (Weakness 1 & 2). Additional Ablation Study of Moving Parts and Design Choices

The submitted paper includes ablation studies such as the impact of MLLM reasoning ability on manipulation policy learning (Figure 3b), the effect of using different training datasets on reasoning ability (Table 4), and the impact of policy head parameters on manipulation accuracy (Table 5). However, based on your valuable comments, we find that some ablation studies are missing. To address this, we explore the impact of different image encoders and LLMs on the reasoning and manipulation capabilities of our proposed framework, while also examining the effect of using different training datasets on manipulation accuracy. Due to space limitations, please refer to global rebuttal Q2, which includes 1) Image Encoder Ablation, 2) Large Language Models (LLMs) Ablation, and 3) The Impact of Training Strategies on Manipulation Abilities.

For evaluation metrics, we select several MLLM benchmarks to comprehensively assess our model’s reasoning capabilities and generalizability, prioritizing those related to robotics. Detailed descriptions can be found in Lines 647-665 of Appendix E. For manipulation, we follow previous works [41, 51, 12] and selected manipulation successful accuracy as the evaluation metric, as described in Lines 240-246 of Section 4.1. Finally, we will include all the aforementioned ablation studies in the final version and conduct more analysis for each moving part.

• (Question 2). Comparisons with structured LLM policy method.

Due to time limitations, we reproduce the more recent VoxPoser method in the SAPIEN environment to evaluate its performance on tasks identical to ours, including 'Open Drawer,' 'Open Door,' and 'Open Faucet.' The manipulation success rates achieved by VoxPoser are 0.19, 0.36, and 0.12, respectively, which are lower compared to our results (i.e., 0.86, 0.73, and 0.30). Notably, while VoxPoser employs motion planning based on affordance and obstacle maps generated by an LLM, it still lacks comprehensive visual scene understanding. In contrast, RoboMamba possesses strong visual scene and robotic reasoning capabilities, conditioned on both image and text inputs. Therefore, RoboMamba demonstrates more robust performance in our experiments.

Reference: [e](Global rebuttal) [f] Real world robot learning with masked visual pre-training

Thanks the authors for clarifying my concerns and adding the simulation experiments for closed-loop policy learning and comparing with VoxPoser. I am happy to raise my score to 6 but suggest the authors add additional comprehensive real-world experiments to have a broader impact.

• (Weakness 1 & Question 1). Detailed inference speed comparison

Thank you for the constructive comments. Inference efficiency is a crucial evaluation metric in robotic manipulation and poses a major challenge for existing Multimodal Large Language Model (MLLM) based policy models. We compare the control frequency (Hz) of our proposed RoboMamba with previous robotic MLLM. All inferences are conducted on the NVIDIA A100 GPU without any quantization or inference speed-up techniques.

Control frequency (Hz). This measures the number of times per second the model can complete inference and generate a manipulation pose. As shown in Table 1 below, RoboMamba-2.7B achieves a control frequency of 9.0 Hz even with the highest input image resolution (336 x 336). These results demonstrate that RoboMamba not only possesses robust reasoning and manipulation capabilities but also maintains efficient inference speed.

Table 1: Comparison of control frequency with previous robotic MLLMs

Meanwhile, we compared the token output speed (tokens/s) with ManipLLM. This indicates the efficiency of Robotic MLLM in performing language output tasks (e.g., task planning or visual question answering). Specifically, we compare the number of tokens output per second by each model for the same question. ManipLLM can generate 133.1 language tokens per second after receiving 336 x 336 input images and a question. RoboMamba-2.7B is more efficient, generating 898.4 language tokens per second. These results demonstrate that our model maintains efficient inference speed when answering robotic high-level reasoning tasks using language responses.

• (Weakness 2 & Question 3). The effect of manipulation head parameters on performance and efficiency

We have investigated the impact of manipulation head parameters on performance by conducting experiments detailed in Lines 625-634 and Table 5 of Appendix C. The findings reveal that the manipulation success rate across the three configurations is similar. This suggests that RoboMamba, due to its sufficient robotic reasoning capabilities, can achieve effective pose prediction skills at a low cost, as the impact of policy head parameters is not sensitive. Meanwhile, as shown in Table 2 below, the effect of manipulation head parameters on efficiency is negligible, since the number of head parameters is very small compared to the overall RoboMamba parameters.

Table 2. The effect of the number of parameters in the manipulation head

• (Weakness 3 & Question 2). Fine-tuning only a small-sized manipulation head on other MLLMs?

In Lines 296-308 and Figure 3 b) of the submission, we have conducted experiments applying the idea of fine-tuning only a small-sized manipulation head to other MLLMs. For convenience, we present the results in Table 3 below, including additional experiments with TinyLLaVA [72]. With the same fine-tuned policy head and training dataset, our RoboMamba-2.7B achieves promising results compared to other MLLMs. These results demonstrate our finding: if MLLMs possess strong robot-related reasoning abilities, they can be efficiently fine-tuned to learn robot manipulation skills. Meanwhile, the Mamba architecture exhibits strong sequence modeling ability [22]; it produces hidden-space output tokens in an autoregressive manner, effectively capturing complex semantic and spatial information in visual scenes. Therefore, RoboMamba can only fine-tune a simple and efficient MLP policy head to convert Mamba’s output tokens into action poses, rather than using a complex policy head for decoding. This significantly enhances RoboMamba's learning efficiency when facing new manipulation categories or tasks.

Table 3. Fine-tuning only a small-sized manipulation head to other MLLMs

• (Weakness 4). The structure of projection layers and the MLPs

Each MLP used in the projection layer, position head, and direction head consists of two linear layers with a ReLU activation function in between. For the projection layer, the input dimension for the first linear layer is B×N×1024, and the output dimension remains B×N×1024, where B and N represent batch size and tokens, respectively. The input dimension for the second linear layer is B×N×1024, and the output dimension is B×N×2560. For the position head and direction head, both have input dimensions of B×N×128 for the first linear layer, and the output dimensions are B×N×128. The input dimensions for the second linear layer are B×N×128, and the output dimensions are B×N×3, collectively forming the 6-DOF poses of the end-effector. We will publish all the code to ensure RoboMamba's reproducibility.

• (Weakness 5). Minor comments

Thank you for your detailed comments; we will fix all the typos in the final version. For example: 1) We will revise Line 13 to: "To further enhance RoboMamba with action pose predictions for robot control in SE(3) space, we explore an efficient fine-tuning strategy using a simple policy head." 2) We will revise Line 100 to: "ManipVQA enhances robotic manipulation with physically grounded information processed by MLLM." 3) In Line 192, we utilize rotation matrices to represent pose direction. 4) We will revise Line 235 to: " As detailed in Appendix E, we describe the key aspects each benchmark considers when assessing models in the field of robotics."

I thank the authors for their answers and clarifications. I think the additional experiments and results presented in their answers make the paper stronger, and they should be included in the final version of the paper. I retain my original rating, suggesting acceptance.

We greatly appreciate your recognition of our work. We will incorporate the additional experiments and detailed module descriptions into the final version. Thank you once again for your valuable time and feedback.