ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning

We sincerely thank the reviewer for your positive and insightful response to our work. In response to the reviewer's concerns about more baseline comparisons and reward ablation, we have added more experiments. We respond to all concerns point by point.

Q1: While the proposed dense reward function works well for certain tasks, it may not be flexible enough for more complex environments.

R1: One of the core contributions of our work is to propose a sparse reward densification method for general visual language manipulation tasks. We believe that the design principles of these four rewards will greatly inspire the reward design for more complex environments. More refined and flexible reward design for complex environments is an important future work.

Q2: Do experiments in a more complicative environment to test the robustness of ReinboT.

R2: We compare ReinboT, OpenVLA and Pi-0 on more challenging real-world tasks. Please see our response (R1) to Reviewer TPwh.

Q3: Do an ablation study on the reward $r_1$ and test it to see whether ReinboT needs the complicative reward design.

R3: The ablation experiment of reward $r_1$ is shown in the table below. The experimental results show that the loss of any part of $r_1$ will lead to a decrease in model performance. Therefore, we do need to explore the characteristics of multi-modal manipulation tasks from various aspects of modal information to better characterize the data quality distribution.

Ablation Study on $r_1$

Q4: Provide the readers with experience on how to set $\lambda$ and $m$.

R4: The original imitation learning loss is still the main supervisory signal for model learning, while the ReturnToGo prediction is more of an auxiliary role. Therefore, the $\lambda$ needs to be appropriately set smaller, and we recommend starting from around 0.001. On the other hand, the $m$ is utilized to control the model’s sensitivity to different expectation levels, thereby adjusting the model’s fitting characteristics for the ReturnToGo distribution. After extensive ablation experiments, we recommend setting the $m$ from around 0.9.

Q5: Why does ReinboT need to predict future image?

R5: Predicting future images is just consistent with the baseline method (GR-1). The core contribution of ReinboT lies in the introduced reward densification method and the organic integration of the RL return maximization principle in VLA.

Q6: Why applying RL in VLA may bring obstacles to transformers? Did you fine-tune VLA with PPO? These results may support the claim more convincingly.

R6: Transformer architectures, especially large pre-trained VLMs, are sensitive to noisy gradients introduced by online RL. Directly fine-tuning the entire VLA model using RL loss can destabilize training. Unlike language tasks, robotics online RL requires exploration, which introduces high-variance updates that can corrupt the carefully tuned parameters of the VLM or even cause it to change catastrophically. Although we do not currently fine-tune VLA directly with PPO, recent research [1] confirms the above. Intuitively, in the online actor-critic RL learning setting, the exploration behavior of the policy is prone to produce some bad trajectory states, and the critic network that has just been trained will have difficulty accurately evaluating the value of these trajectory states, thus misleading the fine-tuning of the VLA policy model and causing performance degradation. In the context of multi-modal manipulation tasks, how to effectively achieve accurate estimation of the critic network and reduce training and memory overhead is also an open question. In contrast, our currently proposed ReinboT bypasses the thorny critic network estimation problem and effectively achieves the benefits of maximizing rewards in RL under the offline learning setting. Further extending ReinboT to the online setting is an important future task.

[1] Guo, Y., Zhang, J., Chen, X., Ji, X., Wang, Y. J., Hu, Y., & Chen, J. (2025). Improving Vision-Language-Action Model with Online Reinforcement Learning. arXiv preprint arXiv:2501.16664.

We sincerely thank the reviewer for your insightful comments, which can further enhance the completeness and thoroughness of our work. In response to the reviewer's concerns about more baseline comparisons, we have added more experiments. We respond to all concerns point by point.

Q1: A direct comparison with Open-VLA and Pi-0.

R1: We further fine-tuned OpenVLA and Pi-0 on more than 900 real-world ur5 robot arm grasping and placing trajectories (including small square wooden blocks, plastic cups, corn, green peppers and other objects). The training configuration is 8xA100 GPUs, batch size is 8 (for OpenVLA) and 32 (for Pi-0), 40000 steps of fine-tuning. The test results show that our proposed ReinboT has competitive performance with Pi-0, with a success rate of about 0.8, which is better than OpenVLA (Only about 0.35). Compared with ReinboT, Pi-0 is pre-trained on a wider and more diverse range of manipulation data and has more model parameters (Pi-0 model parameters are about 3B, while ReinboT has about 300M). Therefore, the capabilities of ReinboT deserve further development, and expanding its training data volume and network parameter scale is an important future work. On the other hand, we speculate that the main reason for the low success rate of OpenVLA is that it only uses a single image observation and a discretized action space. Local perception makes it difficult to grasp the spatial position of objects, and discretized actions also have serious error accumulation problems in long-horizon manipulation tasks, resulting in poor performance.

Q2: How does ReinboT compare in terms of inference speed relative to prior VLA models like OpenVLA?

R2: In our ur5 real machine deployment, the inference speeds of ReinboT, Pi-0 and OpenVLA models are about 0.09s, 0.12s and 0.2s respectively. In fact, ReinboT also uses action chunking technology to generate 64 future actions at a time on the ur5 manipulation task. If we execute multiple steps of action (currently only one step of action is executed by default), the inference efficiency of the model can be further improved. In addition, we did not make any additional designs like Pi-0. The ReinboT model is deployed on the local platform, and the OplenVLA and Pi-0 models are deployed on the remote server. Local deployment platform: CPU: 12th Gen Intel(R) Core(TM) i7-12700F; GPU: NVIDIA GeForce RTX 2060/PCIe/SSE2. Remote server platform: CPU: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz; GPU: NVIDIA A100-SXM4-80GB.

Q3: How does Reinbot perform compared to imitation learning + data curation?

R3: The core idea of combining imitation learning with data curation is to use a metric that measures the distance between source and target data to filter out high-quality data for imitation learning. Therefore, in our VLA problem setting, an intuitive data curation baseline approach is to use ReturnToGo as a trajectory quality metric and filter out high ReturnToGo data for imitation learning. The performance comparison of the proposed ReinboT and the data curation baseline method on CALVIN is shown in the following table. Experimental results show that ReinboT can achieve superior performance. Compared with data management methods that directly discard low-quality data, ReinboT can have a more fine-grained understanding of the distribution of mixed-quality data, thereby better guiding action generation.

Comparison on CALVIN Mixed-quality Data (Test D)

Q4: the reward engineering seems to require some insights from humans, is there any idea to automate the reward engineering?

R4: Previous work used off-the-shelf multi-modal LLM to automatically label robot trajectories with rewards. However, the prompts of LLM still require time and effort to design. The hallucination problem of LLM also significantly affects the accuracy of the designed rewards. In contrast, the rule-based densified reward we proposed is based on the characteristics of long-horizon multi-modal manipulation tasks. It is naturally suitable for manipulation tasks, thus having a wider range of applicability. For new manipulation tasks, we only need to slightly adjust the velocity threshold as the key point criterion and the reward weight used to balance the reward value. This process does not impose an undue burden on researchers' time and energy.

We believe that the reviewer's concerns have been fully addressed. Please convey any additional concerns that hinder the reviewer from improving the score in a timely manner. Thank you again for the reviewer's detailed and thorough review comments.

We sincerely thank the reviewer for your positive response and affirmation of our work. We respond to the reviewer's concerns point by point.

Q1: Experiments: ReinBoT reported a SoTA performance in the CALVIN benchmark. I am not an experienced researcher in VLA area but wanna know why not include more benchmarks like as described in OpenVLA? I think this would much strengthen this paper.

R1: In the field of VLA for general-purpose tasks, model training generally requires a large amount of training data and computing resources, as well as a hardware platform adapted for real-machine deployment. Therefore, conducting experimental comparisons on a wider range of benchmarks is a time-consuming and laborious task that is not affordable for general laboratories. Moreover, in order to provide a direct comparison with previous work (OpenVLA and Pi-0), we further fine-tuned OpenVLA and Pi-0 on more than 900 real-world ur5 robot arm grasping and placing trajectories (including small square wooden blocks, plastic cups, corn, green peppers and other objects). The training configuration is 8xA100 GPUs, batch size is 8 (for OpenVLA) and 32 (for Pi-0), 40000 steps of fine-tuning. The test results show that our proposed ReinboT has competitive performance with Pi-0, with a success rate of about 0.8, which is better than OpenVLA (Only about 0.35). Compared with ReinboT, Pi-0 is pre-trained on a wider and more diverse range of manipulation data and has more model parameters (Pi-0 model parameters are about 3B, while ReinboT has about 300M). Therefore, the capabilities of ReinboT deserve further development, and expanding its training data volume and network parameter scale is an important future work. On the other hand, we speculate that the main reason for the low success rate of OpenVLA is that it only uses a single image observation and a discretized action space. Local perception makes it difficult to grasp the spatial position of objects, and discretized actions also have serious error accumulation problems in long-horizon manipulation tasks, resulting in poor performance.

Q2: For the behavior smoothness reward defined in Eq.(6), it seems to encourage to increase the velocity and acceleration. Why this design leads to smooth actions? Or did you miss a minus sign？

R2: The reward weight $w_3$ corresponding to the reward $r_3$ in formula (6) is a negative number. Therefore, the overall reward is to suppress the velocity and acceleration of the robot manipulation, thereby enhancing the smoothness of the action.

Q3: What are the weights of the four reward components?

R3: The four weights are $\frac{0.1}{T_{traj.}}$, $\frac{0.1}{T_{traj.}}$, $\frac{-0.01}{T_{traj.}}$ and $\frac{0.1}{T_{traj.}}$, where $T_{traj.}$ represents the corresponding trajectory length. The reward weights are summarized in Appendix Table 4.

We sincerely thank the reviewer for your high affirmation and recognition of our work. We respond to the reviewer’s concerns point by point.

Q1: I think the criteria for keypoints in demonstration can be sensitive to the quality of demonstrations; for instance, when demonstrations have mis grasps, there will be false positive keypoints. Some discussions on the sensitivity of quality of demonstrations are warranted for more effective real-world usage.

R1: The designed rule-based densification reward contains four items: sub-goal achievement, task progress, behavior smoothness, and task completion. The task completion reward is result-oriented and is used to determine whether the entire trajectory is successful. The other rewards are process-oriented and are used to assist the robot's decision-making actions. In some cases, false positive keypoints appear when there are incorrect grasping behaviors in the demonstration. However, the behavior smoothness reward suppresses the velocity and acceleration of the robot's body state and actions, thus penalizing such incorrect behaviors of violent back-and-forth grasping. Therefore, the final densified reward can be resistant to such noise in the demonstration data to a certain extent and is not so sensitive to the quality of such demonstrations.

Q2: Comparison against other VLA architectures and RL framework, such as OpenVLA with actor-critic, can help demonstrate the particular combination of the proposed approach.

R2: It is currently an open problem to efficiently combine VLA models such as OpenVLA with the actor-critic framework in RL. This is because OpenVLA discretizes the action space, while the standard actor-critic algorithm models in a continuous action space. Moreover, in the context of multi-modal manipulation tasks, it is also a problem to efficiently achieve accurate estimation of the critic network and reduce training and memory overhead. In contrast, the proposed ReinboT circumvents the thorny critic network estimation problem and effectively implements the benefit of maximizing the return in RL.