Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics

Dear Reviewer 5ayi,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1] Lack of novelty

We clarify that our contribution is NOT in applying the RL training method, GRPO from DeepSeek-R1, to a new domain, but in proposing a novel framework specifically tailored to address the unique challenges for enhancing the embodied reasoning of LVLM. Specifically, our contributions are following three folds:

1. Challenges in Applying RL Algorithms for Improving Embodied Reasoning of LVLM

• Embodied QA lacks definitive answers, leading to suboptimal training targets.
• Current AIs demonstrate poor embodied reasoning performance, so most annotations rely on human experts.

2. Core Innovation: Waypoint-based Embodied Reasoning

• Since the main goal of robot control is action generation, we enhance overall embodied reasoning by predicting waypoints that are fixed targets and easy to achieve.

3. Methodological Advance: MCQA and Auxiliary Task Integration

• Numeric states have limited representational power, making reasoning enhancement through state prediction alone challenging
• We design diverse QA formats (MCQA, open-ended) and validate these approaches through experiments
• To enhance performance, we further consider auxiliary training tasks such as state prediction to improve understanding of numeric states

To summarize, robot-r1 suggests a novel embodied reasoning training framework that enhances performance using numeric state information without any human annotation. Our framework utilizes GRPO, but overcomes many practical challenges stated above while applying it to a robot domain.

---

[W2,W3] Terminology and format error

Thank you for your suggestion. We will clarify the concepts by providing clear definitions of the terms consistently. In particular, we will define these terms:

• “planning” refers to the entire sequence of all high-level actions required to complete a task.
• “high-level action reasoning” refers to linguistic high-level action expression
• “movement reasoning” refers to a more accurate expression that indicates a concrete change in direction within the 3d space, e.g., “move up”, “move down”.

And we will fix the table format in the final draft.

---

[Q1] Is Robot-R1 Bench MCQA-style? How does MCQA-trained Robot-R1 adapt to open-ended benchmarks?

All evaluation environments did not use the MCQA format, and all models were evaluated through open-ended answer generation. This is because Robot-R1 enhances overall embodied reasoning performance by solving MCQA through the reasoning process during training, i.e., the knowledge is easily generalizable to various domains.

---

[Q2] Unfair comparison between Robot-R1 and SFT

Our comparison is fair because we already did an open-ended generation evaluation for both Robot-R1 and SFT, as we also mentioned in our response to [Q1].

---

[Q3] GPT-4o performs better at EmbodiedBench Manipulation

We first clarify that GPT-4o can’t be a fair baseline, as it has a significantly larger model size than Qwen-2.5 VL 7B, which we used to train Robot-R1. As our goal is to develop the training methodology to enhance the embodied reasoning in robotics, our baseline is an initial backbone LLM (no fine-tuning) or other training methods such as SFT. The reason why we present the performance of GPT-4o and other LLMs is to provide a useful reference for better performance judgment.

Additionally, EmbodiedBench is an agent benchmark, requiring not only embodied reasoning but also overall in-context learning capabilities. Meanwhile, Robot-r1 was trained to enhance embodied reasoning performance, not the performance of an LLM agent. Therefore, Robot-R1 is much unfavorable in this benchmark, compared to GPT-4o.

---

[Q4] Computing cost

We have disclosed all the prompt templates used in the experiments in B.2, allowing for easy token calculation. The average prompt length used in RL training is approximately 700 tokens, and that used in SFT is approximately 400 tokens.

The benchmark also includes all prompts publicly. Each problem for evaluation consists of approximately 200 tokens, and the model takes approximately 20 minutes for one A100 to generate all responses for a total of 215 problems. The API cost for LLM-as-judge is approximately $2 for scoring all responses in the benchmark. Each LLM-as-Judge prompt consumes approximately 800 prompt tokens and 90 response tokens.

---

[Q5] Training Robot-R1 with open-ended QA format outperforms MCQA format at training accuracy

We clarify that the results from the open-ended QA format and MCQA format are not comparable in Figure 11(a), because the applied pivot of the y-axis is different between the MCQA and open-ended formats. The open-end QA’s y-axis indicates (1 - L1 loss) while MCQA’s y-axis is QA accuracy.

---

[Q6] How can the accuracy of scoring using large models in the Robot-R1 Bench be ensured?

We first note that prior works in the literature of LLM-as-judge have observed that the evaluation with strong LLMs such as GPT-4 and human evaluation exhibit high correlation [1,2,3].

Nevertheless, to further investigate the validity of this approach for Robot-R1 Bench, we measured the correlation between real human scores and robot-r1 bench scores, by having human experts score responses in the same environment as the LLM-as-judge model scores responses in Robot-R1 Bench.

Specifically, we conducted the test using a subset of 40 problems from the Robot-R1 bench with 8 participants. The test was conducted as a blind evaluation of Robot-R1 and GPT-4o responses, and we defined the median score obtained from this evaluation as the human score.

\begin{array}{c|cccc} \hline & Planning & Subtask & Movement & Spatial \newline \hline Pearson Correlation & 0.3315 & 0.8974 & 0.8931 & 0.8961 \newline \hline \end{array}

The following table shows the Pearson correlation between LLM and human scoring, which supports the reliability of the Robot-R1 bench. Meanwhile, planning shows a somewhat lower correlation coefficient. This is likely because planning tasks typically involve longer responses, which increases the possibility of multiple valid answers. This structural characteristic makes accurate evaluation inherently challenging for planning tasks.

However, Robot-R1 particularly excelled in the other components, and we believe these results do not compromise the reliability of our experimental findings. We will further scale up the reliability assessment during the remaining period and include the results in the final draft.

---

[Q7] Why is RL better than SFT?

While the datasets used in RL training are constructed using specific rules, the model actually learns the embodied reasoning process that emerges throughout the training process, which is automatically generated and optimized.

We also evaluated across diverse environments, including Robot-R1 bench, EmbodiedBench, and SpatialRGPT, which all involve different tasks from our training objective of waypoint prediction. We observed overall performance improvements compared to pre-trained models across these varied tasks. These results experimentally demonstrate the generalization capability of our RL approach.

Meanwhile, research demonstrating that SFT lacks generalization and causes forgetting compared to reassigning incentivized RL methods is a widely known fact in LLM research and has been experimentally proven across a wide range of fields, math, coding, vision etc [4,5]. For example, RL is known to achieve strong generalization performance with minimal data compared with SFT, even with a single sample [6].

---

[1] Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023
[2] Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, EMNLP 2023
[3] Dubois et al., Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, COLM 2024
[4] Chu et al., SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, ICML 2025
[5] Guo et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, Arxiv 2025
[6] Wang et al., Reinforcement Learning for Reasoning in Large Language Models with One Training Example, Arxiv 2025

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

[q1 & q2] Unfair comparison between Robot-R1 and SFT

We first note that SFT has been conventionally trained in an open-ended QA format [1,2] and the majority of evaluations in this paper are conducted in an open-ended format as well. Hence, we believe this open-ended setup is valid and favorable for SFT.

Nevertheless, to address your concern about fairness, we conducted two additional experiments using MCQA-SFT and MCQA-CoT-SFT approaches based on the datasets used for Robot-R1 training. The results are presented in the table below. SFT models trained with the MCQA formats still exhibit significantly lower performance compared to Robot-R1.

Specifically, MCQA-CoT improved planning and spatial reasoning compared to open-ended CoT, but still underperformed even the non-fine tuned model and lost the previously observed primitive movement gains. We will include this result in the final draft.

\begin{array}{c|cccc} \hline \text{Metric} & \text{planning} & \text{subask} & \text{movement} & \text{spatial} \newline \hline \text{Qwen} & 1.66 & 1.04 & 0.58 & 1.40 \newline \hline \text{Robot-R1} &1.44 & 1.30 & 0.76 & 1.51 \newline \hline \text{MCQA-SFT} & 0 & 0.52 & 0.12 & 0.94 \newline \text{MCQA-CoT-SFT} & 1.22 & 1.28 & 0.58 & 0.37 \newline \text{Open-end-CoT-SFT} & 0.60 & 1.28 & 0.70 & 0.29 \newline \hline \end{array}

---

[q7] Why is RL better than SFT?

We appreciate the reviewer’s insightful suggestion. Following your request, we provide intuitive examples to illustrate how Robot-R1 achieves better generalization than SFT w/ CoT. Specifically, we highlight two distinctive reasoning behaviors that emerged from reinforcement learning in Robot-R1, which are not observed in the SFT w/ CoT model:

1. Adaptive Reasoning Planning

• [Robot-R1]: Through reinforcement learning, Robot-R1 learns to adaptively determine the type and score of reasoning needed to solve the current task. This adaptive planning enhances the model to generalize effectively to new tasks.
• [SFT w/ CoT Limitation]: SFT w/ CoT relies on fixed, template-based reasoning patterns designed by human annotators. This rigid structure may fail to accommodate the flexibility required to solve out-of-distribution tasks.
• [Robot-R1 Example]
  • "To determine the movement needed to get to the next keypoint, we need to consider the goal of putting the rubbish into the bin …"
  • “To determine the next waypoint for the robot to reach the button with the maroon base, we need to analyze the distance and alignment from the current position to each of the potential waypoints …”
• [SFT w/ CoT (MCQA) Example]
  • “Plan: 1. Align the gripper by moving it over the base while holding the phone. …”
  • “Plan: 1. Move the gripper to align vertically over the bin and release the gripper. …”

2. Capturing Causal Relationship

• [Robot-R1]: Robot-R1 learns to generate the reasoning steps by explicitly capturing the causal relationships between them. This leads to more coherent reasoning sequences, improving overall consistency and enabling better generalization to new tasks.
• [SFT w/ CoT Limitation]: The training data for SFT w/ CoT is heuristically constructed, which makes it difficult to capture complex causal relationships between reasoning steps linguistically. Consequently, the diversity and quality of causal connections are inherently constrained, hindering the model’s ability to adapt to new tasks.
• [Robot-R1 Examples]
  • “The task is to lift the lid off the pan using the handle. To achieve this, we need to move the robot's arm towards the pan. …”
  • “… The coordinates relative to the robot's current position confirm that moving front and slightly left aligns with the visual information. Therefore, the next step in task planning is moving front and slightly left …”
• [SFT w/ CoT (MCQA) Example]
  • “Plan: 1. Move the gripper to align vertically and position just above the red button. 2. Move the gripper down for pressing the button. Subtask: Move the gripper to align vertically and position just above the red button. …”
  • “Plan: 1. Lift the saucepan lid upward. Subtask: Lift the saucepan lid upward. …”

These elements contribute to enhanced generalization performance when encountering new tasks, and are neither included in SFT data nor easy to construct manually.

---

[1] Zawalski et al., Robotic control via embodied chain-of-thought reasoning, arXiv 2024
[2] Chen et al., Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, CVPR 2024

We are glad that our response has successfully addressed your concerns. To further address your remaining concern regarding CoT-SFT vs. RL, we have prepared additional discussions and experimental results.

1. CoT Quality Concerns: While we agree that putting more effort into SFT dataset construction might yield improvements, we would like to emphasize that a key advantage of the RL approach is its ability to achieve better performance with comparable effort. For instance, Robot-R1 can achieve strong performance using datasets constructed with minimal domain knowledge and without external annotators, relying solely on state information. On the other hand, even our CoT-SFT dataset required more effort to incorporate human-annotated planning and subtask reasoning, yet it performed worse than Robot-R1. We expect that RL performance can be enhanced if we invest effort and resource equivalent to those used for the CoT-SFT dataset.

2. High-Quality CoT Dataset Training: Nevertheless, to address your concerns, we try our best to create higher-quality SFT data and conduct training experiments. To this end, using GPT-4.1-mini, we generate reasoning for the MCQA data used in RL training, conditioned on human-annotated plans and subtask reasoning, with additional filtering based on correctness. As shown in the table below, there are performance gains in subtask and movement categories but still exhibited significant degradation in planning and no improvement in spatial reasoning. This demonstrates that simply improving CoT quality may not fundamentally resolve generalization and forgetting issues.

3. Complementary Relationship Between SFT and RL: As you mentioned, reasoning from high-quality SFT models may surpass that obtained from Robot-R1. However, RL and SFT can be used together, as good SFT models provide effective initialization points for RL training (cold-start RL). To verify this, we conducted Robot-R1 training on the high quality CoT trained SFT model and achieved equal or superior performance across all tasks without performance degradation. This demonstrates the generalization capabilities of RL and the compatibility of both methodologies. The resulting reasoning would likely exceed that obtained from SFT alone.

However, performance gains decreased in some tasks (spatial, movement) compared to Robot-R1 alone, presumably because SFT reduced answer diversity, constraining the RL exploration space. Training SFT with high-quality, diverse data would likely resolve this issue, which I believe is an interesting future direction to explore more.

4. RL Reasoning Reliability: Due to RL's limited control over model’s behavior, it is possible that the model learns incorrect reasoning paths. However, experimental results show this is not severe enough to significantly impair overall generalization capabilities and performance. This issue can be mitigated through cold-start SFT approaches or reasoning path refinement and retraining using SFT.

We will include the above discussion in the final draft and ensure all terminology and formatting errors are corrected.

\begin{array}{c|cccc} \hline \text{Metric} & \text{planning} & \text{subask} & \text{movement} & \text{spatial} \newline \hline \text{Qwen} & 1.66 & 1.04 & 0.58 & 1.40 \newline \text{Robot-R1} &1.44 & 1.30 & 0.76 & 1.51 \newline \hline \text{MCQA-CoT-SFT} & 1.22 & 1.28 & 0.58 & 0.37 \newline \hline \text{High Quality MCQA-CoT-SFT} & 0.98 & 1.34 & 0.88 & 1.4 \newline
\text{Robot-R1 on High Quality MCQA-CoT-SFT} & 1.22 & 1.44 & 0.88 & 1.43 \newline \hline \end{array}

Dear Reviewer 8V4V,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1,W4,Q1] Reward sparsity & QA design

We clarify that our main goal is to enhance various embodied reasoning abilities. For establishing QA design, we compared both MCQA and open-ended QA approaches and experimentally confirmed that MCQA can better enhance embodied reasoning performance. We assume that MCQA provides more opportunities for diverse embodied reasoning through the process of comparing answer candidates, compared to open-ended approaches (see details in Table 6). Based on these results, we chose the MCQA approach despite some potential precision loss.

---

[W2,Q3] Judging bias of LLM-as-judge evaluation: Human cross-validation experiments?

To address your concern, we conducted a human cross-validation experiment on a subset of the Robot-R1 bench. See details in our response to Reviewer 5ayi's [Q6].

---

[W3] Evaluation on real robot setup

Following your constructive suggestion, we conduct an additional experiment by constructing the Robot-R1 Bench on the BridgeV2 real robot video dataset [1] and then evaluating various LVLMs, including Robot-R1 (trained on RLBench). As shown in the below table, we find that Robot-R1 consistently outperforms or is competitive with the baseline, Qwen (non-finetune), indicating Robot-R1 can be successfully transferred to real robot environments, despite the fact that it was trained using simulation data. We will include the results in the final draft.

Here, one can observe that low-level control reasoning is just competitive with the non-finetuned model (Qwen), which we attribute to coordinate system changes. In RLBench, the camera position was always fixed facing the robot front, so we trained the model to describe robot direction and positional relationships based on images. With real robots, the camera position changes, requiring predictions of relative movements based on the robot arm as a reference. We believe this issue could be mitigated if the model were trained based on robot positioning from the beginning, which we plan to address in the final draft.

\begin{array}{c|cccc} \hline & Planning & Subtask & Movement & Spatial \newline \hline \text{Qwen} & 1.20 & 0.89 & 0.46 & 1.53 \newline \text{Gemini-1.5 Pro} & 1.70 & 1.16 & 0.39 & 1.65 \newline \text{Robot-R1} &1.38 & 1.10 & 0.45 & 1.56 \newline \hline \end{array}

---

[Q2] Can the final model be used in the action policy?

The main goal of Robot-R1 is to enhance LVLM's embodied reasoning performance, and it would be challenging to use directly for low-level control due to insufficient precision. However, improved embodied reasoning performance often benefits VLA training [1,2], and we expect that incorporating Robot-R1 into VLA training would yield models with superior performance, which we believe is a promising direction to explore in the near future.

---

[Q5] RLBench is too simple

Thank you for your suggestion. However, we would like to clarify that RLBench contains many complex tasks that require multi-step reasoning, and we select them to construct the training and test datasets. For example, “put rubbish in bin” requests the robot agent to identify the trash, pick it up, carry it to the bin, and finally dispose of it. Furthermore, we already have considered EmbodiedBench Manipulation (see Table 4 in the paper), which includes multi-step reasoning like “Stack the front star on top of the right cylinder”. We also note that we find the improvement from Robot-R1 in more challenging environments, such as the Bridge dataset (see [W3]). Nevertheless, we plan to include more experimental results on various benchmarks that require complex reasoning in the final draft.

---

[Q6] Can this method be extended to the rotation part?

Thank you for the insightful suggestion. Although expressing rotations in language is challenging, we believe we can construct the tasks regarding rotations by using the concept of roll, pitch, and yaw. We think this extension is indeed an interesting future direction for us, and we will include relevant discussion in the final draft.

---

[1] Walke et al., BridgeData V2: A Dataset for Robot Learning at Scale., Arxiv 2023

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

[Q6] Can this method be extended to the rotation part?

For comparison, we prepared experiments on a rotation-containing task: "take_plate_off_colored_dish_rack" as a single task. We examined performance changes when adding an auxiliary task to predict pitch, roll, and yaw for the next waypoint. Results showed that simply adding this component actually degraded performance, suggesting that predicting actions different from existing xyz movements negatively impacts existing performance. Integrating rotation would require separate research, and we believe that it would be an interesting future research direction. \begin{array}{c|cccc} \hline \text{Metric} & \text{planning} & \text{subask} & \text{movement} & \text{spatial} \newline \hline \text{Robot-R1 w/o rotation} &1.66 & 1.4 & 0.56 & 1.52 \newline \hline \text{Robot-R1 w rotatotin } & 1.6 & 1.34 & 0.44 & 1.55 \newline \hline \end{array}

We will include the above discussion in the final draft

We first apologize for the missing references. The statement "However, improved embodied reasoning performance often benefits VLA training" corresponds to the following references: [2,3].

---

[Q2] Can the final model be used in the action policy?

To address your concern with concrete evidence, i.e., verifying whether the hypothesis actually applies to our model, we trained both Qwen2.5-VL and Robot-R1 as VLAs using the Groot-n1.5 [4] VLA training framework. We conducted experiments on the widely validated LIBERO [5] dataset for VLA training, using libero-{10, goal, object, spatial} data for 60K training steps. The results in the below table demonstrate that Robot-R1 achieved superior performance compared to the non-fine-tuned baseline model.

Specifically, we observed high performance gains in Libero Goal, which requires complex operations rather than simple pick-and-place, aligning well with Robot-R1's embodied reasoning enhancements. While the Objective category showed slight performance degradation, we assume this is due to RLBench's limited object diversity during Robot-R1's training.

\begin{array}{c|ccccc} \hline \text{Metric} & \text{LIBERO-10} & \text{LIBERO-Goal} & \text{LIBERO-Objective} & \text{LIBERO-Spatial} & \text{Avg} \newline \hline \text{Qwen2.5 VL 7b} & 44.0 & 61.6 & 83.4 & 86.8 & 69.0 \newline \hline \text{Robot-R1} & 45.8 & 73.8 & 80.4 & 87.0 & 71.8 \newline \hline \end{array}

[2] Chen et al., Training Strategies for Efficient Embodied Reasoning, arXiv 2025
[3] Yang et al., Magma: A foundation model for multimodal AI agents, CVPR 2025
[4] Bjorck et al., Gr00t n1: An open foundation model for generalist humanoid robots, arXiv 2025
[5] Liu et al., Libero: Benchmarking knowledge transfer for lifelong robot learning, NeurIPS 2023

---

[Q5] RLBench is too simple

As you suggested, we explored RoboCerebra, but determined it is currently unavailable as the code and required VLA for evaluation are not publicly available. Regarding VLABench, we initially considered that it is inappropriate for Robot-R1 evaluation due to:

1. its use of multi-view images split into quadrants as a single input, along with additional annotated mask images, which differs significantly from Robot-R1's single front-view natural image training
2. its inclusion of reasoning elements unrelated to manipulation

Nevertheless, to address your concerns, we evaluated Robot-R1 on the VLM evaluation component of VLABench. As shown in below table, performance improvements were observed in M&T (Mesh and Texture) and Spatial categories, while performance degradation occurred in PhysicalLaw and Complex categories. However, we note that PhysicalLaw and Complex are distant from our target tasks, as they require reasoning unrelated to manipulation.

[PhysicalLaw task examples]

• "We have three objects with different densities. Choose the object with the smallest density..."
• "We are going to test the friction of different objects. Choose the object that slides the slowest on a steel ramp."

[Complex task examples]

• "Please rearrange these books in chronological order starting from the first published to the latest published ones."
• Texas Hold'em scenario: "Secure the most formidable hand in this play."

One can observe that these tasks differ significantly from Robot-R1's focus on learning movement design for task execution. Improving performance on such tasks would require joint RL training on Steam or common knowledge-related data. On the other hand, the categories showing improvement, particularly Spatial, are closely related to action planning and include tasks requiring multi-step manipulation reasoning. M&T contains questions involving simple manipulation.

[Spatial examples]

• "Add 2nd condiment from bottom to the dish."
• "Please take the third book from the left on the top layer."

[M&T examples]

• "Please take the book one_hundred_years_of_solitude"
• "Add salt to the dish"

The performance improvement in Spatial demonstrates that Robot-R1 can effectively handle complex manipulation tasks despite significant environmental differences in image input. This is also significant in that it achieves performance gains while incurring minimal performance degradation, even when all SFTs fail.

\begin{array}{c|cccccc} \hline \text{Metric} & \text{M\&T} & \text{CommonSense} & \text{Semantic} & \text{Spatial} & \text{PhysicalLaw} & \text{Complex}\newline \hline \text{Qwen2.5 VL 7b} & 39.02 & 40.78 & 37.64 & 35.83 & 39.20 & 38.51 \newline \hline \text{Robot-R1} & 46.58 & 41.5 & 37.62 & 40.33 &25.33 & 33.87 \newline \hline \text{CoT-SFT} & 0 & 0 & 0 & 0 & 0 & 0 \newline \hline \end{array}

[Q1]

1. Difficulty of direct comparison

GR00t-n1.5 is obtained through a multi-stage process: 1) NVIDIA trains Eagle2.5-VL[1] from Qwen2.5-Instruct on large-scale video data, and then 2) enhancing grounding capabilities and physical understanding using additional datasets. Lastly, 3) it performs VLA pre-training on robot datasets. Since Eagle2.5-VL already outperforms Qwen2.5-VL and it is undisclosed, a direct one-to-one comparison with Robot-R1 is not straightforward.

For example, Robot-R1 is trained with far fewer data, about 2.5K image. However, GR00t-n1.5 uses massive embodied datasets (e.g., the OpenX Embodiment [2] dataset alone, which accounts for 27.3% of its pre-training, is around 13 TB).

2. Planned SFT comparison

To address your question, we plan to add a direct comparison experiment with an SFT-based VLA in the final draft, to show the superiority of our RL-based VLA. However, we remind you that SFT has shown limited generalization and forgetting issues compared to RL; for example, CoT-SFT was not well-generalized on VLABench unlike Robot-R1. Hence, we expect that the performance under CoT-SFT would also be limited or possibly negative.

3. Research focus and VLA integration

We first clarify that our research aims not to train VLA, but to enhance the embodied reasoning capability of VLMs. One the side, prior work such as Hi-Robot [3] has shown that generating high-level actions first, followed by hierarchical low-level action generation, can substantially boost performance. We expect that this kind of better VLA approach may fully exploit Robot-r1’s strengths under VLA framework similar to those in Pi0.5 [4] and GR-3 [5].

---

[Q2]

We believe that tasks requiring high-level action generation tend to exhibit smaller domain gaps than those relying heavily on low-level perception, since conceptual action space in text form is likely to remain more consistent across domains than raw visual input. This may explain why VLA performance improved the most on complex tasks such as LIBERO-goal and why, in Robot-R1 Bench built from BridgeV2, subtask reasoning was transferred most effectively.

Overall, we really appreciate your time and effort actively spent in the discussion period. We will incorporate all of our discussions in the final draft, which we do believe would further strengthen our paper!

---

[1] Chen et al., Eagle 2.5: Boosting long-context post-training for frontier vision-language models, arXiv 2025
[2] O’Neill et al., Open x-embodiment: Robotic learning datasets and rt-x models, ICRA 2024
[3] Shi et al., Hi robot: Open-ended instruction following with hierarchical vision-language-action models, arXiv 2025
[4] Physical Intelligence et al., π₀.₅: A vision-language-action model with open-world generalization, arXiv 2025
[5] Cheang et al., GR-3 Technical Report, arXiv 2025

Dear Reviewer 4LvH,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1-1] Lack of further analysis

The evaluation benchmarks used in our paper, Robot-R1 bench, EmbodiedBench Manipulation, and SpatialRGPT, are significantly different from our training task of waypoint prediction. SpatialRGPT even uses completely different inputs based on real images. Robot-R1 demonstrates consistent performance improvements across all these environments, while SFT shows significant performance degradation in certain tasks (e.g., EmbodiedBench). This experimentally demonstrates the generalization capability of RL and the forgetting issues inherent in SFT.

Furthermore, it is well-established in the LLM community that SFT causes catastrophic forgetting, while RL achieves better generalization with minimal data [1]. For instance, Deepseek-R1 showed that models trained with RL on coding and mathematics tasks also improve performance across other domains [2]. Additionally, research has demonstrated that RL can achieve generalization even with single samples [3].

---

[W1-2] RL training is good at generalization?

We would like to clarify that Robot-R1 is trained to explicitly perform embodied reasoning before answering the multiple-choice question. As a result, Robot-R1 generates various embodied reasoning, e.g., planning how to perform the task.

Here, we hypothesize that wrong embodied reasoning leads to wrong responses to multiple-choice questions, and vice versa. As the model is trained to generate a correct response, we can expect that the model can generate more accurate embodied reasoning from the training. We have included the qualitative results regarding the generated embodied reasoning quality in Appendix D.

---

[W2] Limitations in the evaluation

[Validation of Robot-R1 bench] To address your concern, we conducted a human cross-validation experiment on a subset of the Robot-R1 bench. See details in our response to Reviewer 5ayi's [Q6]. In particular, the high correlation observed in items related to low-level control reasoning reinforces our claim that Robot-R1 "surpasses GPT-4o on reasoning tasks related to low-level action control."

[Comparison with GPT-4o] We clarify that GPT-4o can’t be a fair baseline, as it has a significantly larger model size than Qwen-2.5 7B, which we used to train Robot-R1. As our goal is to develop the training methodology to enhance the embodied reasoning in robotics, our baseline is an initial backbone LLM (no fine-tuning) or other training methods such as SFT. For this reason, we present the performance of GPT-4o and other LLMs to provide a useful reference for better performance judgment.

---

[Q1] Cold Start SFT

We could design experiments to perform reinforcement learning on top of an SFT model that inherits Robot-R1's reasoning capabilities. We will conduct these experiments during the remaining period and aim to include the results in the final draft.

---

[Q2] Is it possible to observe a gradual increase in the length of generated tokens during RL training?

We actually observed the opposite trend that the length of the generated token is decreasing during RL training. This occurred since sentences became more concise and shifted from overview-style to descriptive formats. For specific length changes, please refer to Figure 11, and for qualitative analysis, please see Appendix C.

---

[Q3] Has any decontamination been carried out to ensure that the training dataset does not overlap with Robot-R1 Bench? If so, what methodology was employed?

Since RLBench generates new demonstrations each time, we separated the data by extracting from differently generated demos to ensure no overlap between training and evaluation datasets.

---

[1] Chu et al., SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, ICML 2025
[2] Guo et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, Arxiv 2025
[3] Wang et al., Reinforcement Learning for Reasoning in Large Language Models with One Training Example, Arxiv 2025

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Furthermore, we provide intuitive examples to illustrate how Robot-R1 achieves better generalization than CoT-SFT. Specifically, we highlight two distinctive reasoning behaviors that emerged from reinforcement learning in Robot-R1, which are not observed in the CoT-SFT model

1. Adaptive Reasoning Planning

• [Robot-R1]: Through reinforcement learning, Robot-R1 learns to adaptively determine the type and score of reasoning needed to solve the current task. This adaptive planning enhances the model to generalize effectively to new tasks.
• [CoT-SFT Limitation]: CoT-SFT relies on fixed, template-based reasoning patterns designed by human annotators. This rigid structure may fail to accommodate the flexibility required to solve out-of-distribution tasks.
• [Robot-R1 Examples]
  • "To determine the movement needed to get to the next keypoint, we need to consider the goal of putting the rubbish into the bin …"
  • “To determine the next waypoint for the robot to reach the button with the maroon base, we need to analyze the distance and alignment from the current position to each of the potential waypoints …”
• [CoT-SFT Examples]
  • “Plan: 1. Align the gripper by moving it over the base while holding the phone. …”
  • “Plan: 1. Move the gripper to align vertically over the bin and release the gripper. …”

2. Capturing Causal Relationship

• Robot-R1: Robot-R1 learns to generate the reasoning steps by explicitly capturing the causal relationships between them. This leads to more coherent reasoning sequences, improving overall consistency and enabling better generalization to new tasks.
• CoT-SFT Limitation: The training data for CoT-SFT is heuristically constructed, which makes it difficult to capture complex causal relationships between reasoning steps linguistically. Consequently, the diversity and quality of causal connections are inherently constrained, hindering the model’s ability to adapt to new tasks.
• [Robot-R1 Examples]
  • “The task is to lift the lid off the pan using the handle. To achieve this, we need to move the robot's arm towards the pan. …”
  • “… The coordinates relative to the robot's current position confirm that moving front and slightly left aligns with the visual information. Therefore, the next step in task planning is moving front and slightly left …”
• [CoT-SFT Examples]
  • “Plan: 1. Move the gripper to align vertically and position just above the red button. 2. Move the gripper down for pressing the button. Subtask: Move the gripper to align vertically and position just above the red button. …”
  • “Plan: 1. Lift the saucepan lid upward. Subtask: Lift the saucepan lid upward. …”

These elements contribute to enhanced generalization performance when encountering new tasks, and are neither included in SFT data nor easy to construct manually.

We hope that the above statistical evidence and intuitive explanations could resolve your concerns on Robot-R1 (or RL) vs. SFT, However, if you have any other suggestions, we are also happy to incorporate during the discussion period !

Dear Reviewer vxzH ,

We sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.

---

[W1,Q1] Reproducibility

We agree that online reinforcement learning can exhibit variance. To address your concern, we conducted two additional experiments with different random seeds. The results are as follows, where we confirm that our method still demonstrates significant performance improvements over the baseline. We will include this result with error bars in the final draft.

\begin{array}{|l|c|c|c|c|} \hline Method & Planning & Subtask & Primitive &Spatial \newline \hline \text{Qwen} & 1.66 & 1.04 & 0.58 & 1.40 \newline \text{GRPO seed=1} & 1.44 & 1.30 & 0.76 & 1.51 \newline \text{GRPO seed=2} & 1.70 & 1.46 & 0.64 & 1.66 \newline \text{GRPO seed=3} & 1.56 & 1.50 & 0.54 & 1.60 \newline \text{GRPO avg} & 1.57 & 1.42 & 0.65 & 1.59 \newline \hline \text{RLOO} & 1.56 & 1.54 & 0.68 & 1.52 \newline \text{Reinforce++} & 1.40 & 1.08 & 0.62 & 1.38 \newline \hline \text{GRPO with Half dataset} & 1.56 & 1.50 & 0.54 & 1.60 \newline \text{GRPO with single task} & 1.52 & 1.28 & 0.54 & 1.58 \newline \hline \text{Qwen 3B} & 1.66 & 1.04 & 0.58 & 1.40 \newline \text{Qwen 3B + Robot-r1} & 1.58 & 1 & 1.08 & 1.23 \newline\hline \end{array}

---

[W2,Q2] Evaluation on diverse tasks and real robot system

To address these concerns, we created a new Robot-R1 bench based on BridgeV2 real robot data and evaluated our model's performance on it. The results showed that performance improvements remained prominent across high-level actions, indicating Robot-R1 can be successfully transferred to real robot environments, despite the fact that it was trained using simulation data, RLBench. We will include the results in the final draft.

Here, one can observe that low-level control reasoning did not transfer well, which we attribute to coordinate system changes. In RLBench, the camera position was always fixed facing the robot front, so we trained the model to describe robot direction and positional relationships based on images. With real robots, the camera position changed, requiring predictions of relative movements based on the robot arm as a reference. We believe this issue could be mitigated if the model were trained based on robot positioning from the beginning, which we plan to address in the final draft.

\begin{array}{c|cccc} \hline & Planning & Subtask & Primitive & Spatial \newline \hline \text{Qwen} & 1.20 & 0.89 & 0.46 & 1.53 \newline \text{Gemini-1.5 Pro} & 1.70 & 1.16 & 0.39 & 1.65 \newline \text{Robot-R1} &1.38 & 1.10 & 0.45 & 1.56 \newline \hline \end{array}

---

[W3] Dataset size issue

We would like to clarify that models trained with reasoning incentivized RL can generalize effectively with less than 10K training data [1,2], and a recent study has even shown that generalization is possible from a single example [3].

In addition, we find that Robot-R1 trained with a 7.5K QA dataset shows consistent performance gains across diverse tasks from different benchmarks (Robot-R1 Bench, EmbodiedBench Manipulation, and SpatialRGPT-Bench), which empirically demonstrates that the 7.5K QA dataset is sufficient to learn generalizable reasoning capability.

---

[W4] Traning algorithm comparison

We agree with the reviewer in that investigating the proper RL algorithms for embodied reasoning tasks is an interesting problem. However, actor-critic RL like PPO requires excessive GPU VRAM and hence it’s hard to apply in the current constrained resources. Instead, we conduct an experiment by training Robot-R1 with widely used RL algorithms, RLOO [4] and Reinforce++ [5].

As shown in the above table in [W1,Q1], Robot-R1 trained with RLOO achieves similar performance to Robot-R1 trained with GRPO, and Reinforce++ degrades the performance. We think this is because the unstable training of Robot-R1 with Reinforce++, due to weaker reward normalization (batch reward normalization).

We will include this discussion in the final draft.

---

[W5] The approach requires access to a task-specific QA based dataset, which is not available for many embodied applications.

The main goal of Robot-R1 is to enhance VLM's embodied reasoning performance for robotic applications. The training data is constructed based on expert demonstration state trajectories, which can be readily obtained from most robotic learning datasets.

These reasoning abilities can be easily transferred to various tasks, including EmbodiedBench, real robot experiments [W2, Q2], and other benchmarks. This experimentally demonstrates that our approach is not heavily constrained by task-specific datasets.

---

[Q3-1] Scaling law: How does performance scale with dataset size, number of tasks, and model parameters?

We conducted experiments involving three modifications: reducing the training data size by half, utilizing a single task (put rubbish in bin) for training, and downsizing the model from 3B parameters.

As shown in the above table in [W1,Q1], the results demonstrated that Robot-R1 generalizes effectively even with smaller models. When the data volume was reduced by half, performance remained comparable within the margin of error. However, when limiting the number of tasks to single, we observed a slight performance degradation in subtasks. As mentioned in [W3], we attribute these results to the robust generalization capabilities of RL training.

---

[Q3-2] Computing cost: What are the computational requirements compared to SFT?

Training a model with RL (GRPO) requires around 1.5 times more training time than SFT.

---

[Q4] Can you provide more analysis of the failure cases of the proposed approach?

As mentioned in [W2,Q2], the low-level control reasoning capabilities learned from RLBench showed limited transferability to other robotic systems. This limitation is presumed to arise from the discrepancy between the coordinate systems used during the training phase and those employed during the testing phase.

---

[Q5] Judging bias of LLM-as-judge evaluation: Human cross-validation experiments?

Thank you for pointing this out. To address your concern, we conduct a human cross-validation experiment on a subset of the Robot-R1 bench. See details in our response to Reviewer 5ayi's [Q6].

---

[1] Zeng et al., Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, Arxiv 2025
[2] LMMS-Lab, Multimodal Open R1 8K Verified, Hugging Face, 2024/2025
[4] Wang et al., Reinforcement Learning for Reasoning in Large Language Models with One Training Example, Arxiv 2025
[5] Ahmadian et al., Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, Arxiv 2024
[6] Hu et al., Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, Arxiv 2025

---

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors