We sincerely appreciate the reviewer’s supportive comments on our strengths across the quality, novelty, impact, and visualization. Below, we address your remaining concerns clearly and thoroughly.

Q1: Extend to more VLA models and more complex tasks

Thank you for raising this important concern. While our original submission focused on a single model (OpenVLA) and task (pick-and-place) to establish a controlled foundation for comparison, we have since expanded our study in two complementary directions to reinforce the generality of our findings.

First, to evaluate broader VLA architectures, we conducted additional experiments on OpenVLA-OFT [3], a stronger variant of OpenVLA that incorporates action chunking. This model poses increased challenges for RL training due to its higher-dimensional, temporally extended action space. We adapted our RL pipeline accordingly—modifying reward computation, clip ratio, and advantage estimation—and observed consistent improvements from RL fine-tuning over SFT. These results indicate that the benefits of RL are not confined to the original model and extend to more complex action representations.

Second, to evaluate more complex tasks, we introduced a more challenging open-cabinet manipulation benchmark. This task requires more dexterous manipulation with a articulated object, including handle grasping and door opening under significant semantic and geometric variability. Across six out-of-distribution splits—including novel objects, diverse cabinet configurations, and perturbed initial states—RL-fine-tuned policies again consistently outperformed SFT, especially on execution and semantic generalization. This echoes the trend observed in the pick-and-place setting and provides further evidence that RL enhances robustness across manipulation tasks.

While each extension targets a different axis (model or task complexity), both strengthen our core conclusion: RL fine-tuning offers more robust generalization than SFT, particularly in execution and semantic variation, and this advantage holds across models and tasks of increasing complexity.

Q2: Extended to more RL algorithms like SAC

We sincerely thank the reviewer for this thoughtful suggestion. We agree that evaluating our approach with a standard off-policy algorithm such as SAC would be a valuable addition to the study and could further enrich our comparative analysis.

That said, we would like to emphasize that our current comparisons among PPO, GRPO, and DPO still offer meaningful contributions. GRPO and DPO represent recent advances specifically designed to fine-tune policies from demonstrations and preferences—both widely discussed in the context of language and vision-language models. Our finding that PPO, an on-policy RL algorithm, outperforms GRPO and DPO on embodied multi-step tasks is not merely expected—it provides a concrete empirical validation that these methods may fall short when applied to sequential VLA control tasks. To our knowledge, this is the first systematic comparison of these algorithms on such models, and the insights gained are directly actionable for researchers adapting RL methods to VLA domains.

Regarding the inclusion of SAC, we acknowledge its relevance and have actively considered it. However, applying SAC to our setting is non-trivial, as OpenVLA operates in a discrete, high-dimensional action space: 7 dimensions with 256 bins each. This results in an effective action space of 256⁷ combinations, making traditional continuous-action SAC inapplicable without substantial adaptation. One workaround involves Gumbel-Softmax-based relaxation to approximate gradients for discrete actions, but this results in a 1792-dimensional action input to the Q-network, posing severe challenges for learning a stable and expressive critic.

Despite these challenges, we are working diligently to explore SAC-based experiments within the remaining rebuttal/discussion period and will incorporate complete results in a revised version of the manuscript. We appreciate the reviewer’s suggestion and view it as a valuable direction for future iterations of this work.

Q3: Implication of emojis

Thank you for pointing this out, and we apologize for any confusion caused by the use of emojis in Figure 1. Our intent was not to suggest the absolute performance drop of RL in each OOD setting, but rather to highlight the relative performance of RL compared to SFT across the three axes. Specifically, the emojis were chosen to reflect: in out-of-distribution tests, RL yields substantial gains in Execution, moderate improvements in Semantics, and performance on par with SFT for Vision.

We used distinct emojis to visually summarize this pattern of relative advantage, not to indicate differences in absolute performance degradation. We appreciate the reviewer’s feedback and will revise the figure caption and visual elements in the revised version to make this intent clearer.

Q4: Advantage computation in PPO

We did NOT recompute advantages after the first pass. In PPO, a rollout is collected with the current policy, GAE is computed once with the value-function of that policy, and the resulting advantages are kept fixed while the same data are reused for several mini-batch updates across later epochs. This is also how many well-known RL codebases, like Stable-Baselines3, MAPPO and "37 Implementation Details of PPO" blog, handle PPO updates, so our implementation follows common practice.

Q5: Comparison between ours and RT2's Fig4 experiments

Although both our study and the RT-2 experiments in Fig. 4 evaluate policies under several OOD settings, the research questions diverge:

• Our focus is to ask "What unique benefits can RL bring to VLA generalization compared to SFT?". Hence, we start from the same pretrained VLA backbone and compare an RL-fine-tuned policy with its SFT counterpart.
• RT-2’s focus is to show that web-scale pre-training endows VLA models with broadly transferable visual and semantic concepts. Their evaluation therefore contrasts different model architectures, data sources, and training pipelines.

Because of these distinct aims and setups, the two studies are complementary rather than directly comparable. Moreover, our benchmark spans three orthogonal generalization dimensions, offering a more systematic evaluation suite than RT-2's three unseen tests.

We sincerely thank the reviewers for their valuable feedback and positive evaluation of our research. Below, we provide detailed responses to each of your comments.

Q1: Meaning of our PPO recipe and experiments

Thank you for your attention to the desciptions devoted to PPO and Section 4.2. We understand the concern and appreciate the opportunity to clarify its role in the paper.

When we conducted this study, applying PPO to finetune a VLA model remained a non-trivial and open problem. Although there have been some emerging concurrent work [1,2] on PPO+VLA since our submission, our work is one of initial studies in this direction. There was no established or off-the-shelf RL method that could directly be used to study generalization in our study. As such, developing an effective PPO training recipe was necessary to enable a systematic and fair comparison of generalization between RL and SFT. Without this foundation, it would be difficult to make credible claims about RL’s potential contributions to generalization.

Moreover, the training decisions described in Section 4.2 are not standard or trivial; through empirical tuning, we identified several key design choices that significantly impact performance and training efficiency. These insights are valuable in their own right and can guide future work on applying RL to VLA models.

Besides, the effects of these design choices are not isolated to in-distribution performance but show consistent relative performance for out-of-distribution tests. Thus, we base the rest of our comparative experiments on this PPO recipe.

We will make these motivations and the connection to generalization more explicit in the revised version to improve clarity and coherence.

[1] Interactive Post-Training for Vision-Language-Action Models. arXiv:2505.17016

[2] VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning. arXiv:2505.18719

Q2: Is it realistic and practical to apply visual noise as done in our experiments?

Thank you for raising this concern. Stress-testing vision with deliberately extreme textures or lighting is already standard practice for gauging a manipulation policy’s perceptual invariance.

• Purpose-built stress tests are widely accepted. The DMControl generalization benchmark [3] suite overlays randomly sampled textures and colours on every frame to create out-of-distribution (OOD) views; many works like [4] evaluate policy performance under these texture-shifted test sets, treating them as a proxy for unexpected backgrounds and camera disturbances.
• Real-robot studies adopt the similar idea. Prior work on manipulation like [5] evaluates the same policy after changing table, background colours, camera viewpoints, and dynamic Disco lighting conditions, demonstrating that such appearance swaps are considered a practical robustness check.
• Community benchmarks adopt similar disturbed evaluation. The Colosseum benchmark [6] includes “Texture Swap” and “Lighting Perturbation” splits in which object and table surfaces, as well as global illumination hues, are randomised at test time; leading methods are ranked by how gracefully they handle these overlays.

Our Dynamic-Texture and Dynamic-Noise splits follow the same philosophy: although the specific patterns may be unlikely to appear in daily operation, they create a controlled yet harsh visual perturbation that reveals whether a policy has learned stable, object-centric features rather than over-fitting to background colour or illumination. Therefore, the evaluation protocol is both practical and comparable to widely adopted benchmarks.

[3] Generalization in reinforcement learning by soft data augmentation. ICRA 2021

[4] Generalizable Visual Reinforcement Learning with Segment Anything Model. arXiv:2312.17116

[5] Learning to Manipulate Anywhere: A Visual-Generalizable Framework for Reinforcement Learning. CoRL 2024

[6] The Colosseum: A Benchmark for Evaluating Generalization for Robotic Manipulation. RSS 2024

Q3: The quality of the SFT dataset

Thank you for your insightful question. We acknowledge the quality limitation of our SFT dataset as mentioned in the paper. To address this concern, we have taken steps to improve data diversity by tuning our motion planner to explore a larger state space. Additionally, as noted in Section 4.2 (lines 213-214), we experimented with combining motion-planned data with demonstrations collected using a pretrained policy (e.g., Octo). However, our experiments revealed that the performance difference between motion-planned and Octo-collected data was minimal for SFT. Therefore, considering scalability advantages, we adopted the motion planner approach to collect up to 64k trajectories.

Regarding the performance drop after 16k trajectories in Figures 6a and 6b, we hypothesize that with larger datasets, it becomes increasingly challenging for the policy to effectively cover all training samples, potentially leading to the observed performance degradation. It's worth noting that while the success rate shows a decline, the grasp accuracy and continuous grasp accuracy metrics appear to plateau, suggesting that the policy maintains certain capabilities despite the overall performance drop. This phenomenon warrants further investigation, which we plan to investigate further in future work.

Thank you for your insightful comments and for highlighting these important considerations. We appreciate the opportunity to further clarify the goals and contributions of our paper.

We acknowledge that real-world RL for VLA and advanced versions with flow matching are still in early development stages. Our focus, therefore, is not on achieving SOTA performance, but on providing a thorough comparative analysis of RL versus SFT that offers insights applicable across different methods and models.

To this end, we strategically chose a simulation-based evaluation for two compelling reasons:

• Rigorous benchmarking: Large-scale controlled simulations are essential for comprehensive ablation studies, reproducibility, and broad scenario coverage—critical requirements for establishing a benchmark with lasting research impact. Real-world trials alone are hard to provide this level of systematic analysis.
• Intrinsic merit: Simulation-based RL training has its own value, especially given ongoing debates regarding whether simulation-based training can enhance VLA's abilities. Concurrent works [1,2] have also demonstrated the potential advantages of on-policy RL trained in simulated environments. To address further concerns, we have included preliminary real-world results validating sim-to-real transfer during this rebuttal period (detailed in our Q1 response).

To support the generalizability of our findings, we have conducted extensive additional experiments, including tests with alternative VLA architectures, more complex tasks, and off-policy RL algorithms. These results consistently validate our core findings about the distinct effects of RL in comparison to SFT.

[1] Interactive Post-Training for Vision-Language-Action Models. arXiv:2505.17016

[2] VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning. arXiv:2505.18719

Below, we will address your specific questions.

Q1: Additional experiments to verify the conclusions on a real-robot

As discussed, our experiments are intentionally conducted in simulation for scalability and reproducibility when comparing RL and SFT. Nonetheless, we fully agree that real-world performance is a critical aspect to validate and have therefore included a preliminary real-world experiment to assess sim-to-real generalizability—adding an additional generalization dimension to our evaluation.

According to our real-world setups, we adapted the simulation by replacing the WidowX arm with a Franka Panda and aligning the camera views. Using the same pipeline as in our main experiments, we re-trained VLA models with both RL and SFT under this new configuration and deployed them zero-shot on the real robot. This setting introduces a challenging OOD setting involving kinematic differences between the simulated and real arms, varied backgrounds, and different objects (including pepper, cucumber, banana, mangosteen, kiwi, and sponge) and receptacles (in specific, a green bowl). We report the average performance over 30 real-world trials in this table:

As shown in the table, RL fine-tuning in simulation can also help the sim-to-real generalization compared to SFT. We observed that when approaching the object, SFT often exhibits excessive movement and tends to overshoot, resulting in a suboptimal grasping pose and subsequent grasp failures. In contrast, RL, despite showing some jitter in its actions, is able to iteratively adjust its grasping pose around the object. This leads to a higher grasp success rate.

Q2: Additional experiments with a standard off-policy RL algorithm (e.g. SAC)

We sincerely thank the reviewer for this suggestion. We agree that evaluating our approach with a standard off-policy algorithm like SAC would further strengthen our work.

However, applying SAC to OpenVLA is non-trivial, especially within the limited rebuttal period. Our VLA model features a discrete, high-dimensional action space (7 dimensions, each with 256 bins), requiring techniques like the Gumbel-Softmax trick for differentiability. This results in a very high-dimensional (256 × 7) action input for the Q-network, making stable learning particularly challenging.

Despite these technical hurdles, we are trying our best to conduct these experiments and hope to provide these additional results before the discussion period concludes. We are fully committed to incorporating the complete results and a detailed analysis of the SAC experiments into a revised version of our paper. We appreciate the reviewer's valuable feedback to further complement our study.

Q3: Additional experiments with a more advanced VLA model

Thank you for raising this insightful question regarding the choice of our base model. We agree that advanced VLA architectures, such as Pi-0, which incorporate diffusion or flow-matching policies, indeed pose unique challenges for RL fine-tuning. To date, end-to-end RL methods for models like Pi-0 have not been convincingly demonstrated in the literature, and developing such methods would extend beyond the scope of our systematic comparative study.

Nevertheless, we acknowledge the importance of evaluating our conclusions with a more advanced VLA architecture. To address your concern, we have introduced an additional model, OpenVLA-OFT [3], which enhances the original OpenVLA model by integrating action chunking. To adapt our RL pipeline for OpenVLA-OFT, we made the following modifications:

1. Action Chunking: We utilized an action chunk size of 4.
2. Warm-up Model Training:
   • Collected 400 trajectories via motion planning without action filtering.
   • Trained a warm-up model from Haozhan72/Openvla-oft-SFT-libero10-trajall (a checkpoint from Huggingface).
3. RL Fine-tuning Adjustments:
   • Each observation predicts a sequence of 4 actions, with cumulative rewards.
   • To accommodate the higher dimensional action space, we modified the PPO clipping mechanism to perform action-dimension-wise clipping and reduced the clip ratio from 0.2 to 0.1, and set gamma to 0.96 and lambda to 0.85 for stability.

Results show that RL maintains its advantage over SFT even with this more advanced architecture:

[3] Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. RSS 2025.

Q4: Additional experiments on more complex tasks

Thank you for highlighting the need for a task that demands richer dexterity than pick‑and‑place. To address this, we extended the benchmark with an open‑cabinet task that requires grasping a handle and opening the door past 20 °. The reward function assigns 0.1 when the handle is first grasped, 0.1 when the grasp is sustained for five consecutive steps, and 1.0 upon successful door opening. During training, we randomized:

• Vision: 16 backgrounds from the original assets.
• Semantics: 8 cabinets of varying sizes, names, handle types (bar, boxed) and handle positions (up, down, left, right).
• Execution: Cabinet frame translations (±8 cm) and rotations (±π/48).

We then test on six OOD splits: (i) 5 new backgrounds, (ii) whole‑scene dynamic noise, (iii) 4 novel articulated objects (e.g., microwave, mini‑fridge), (iv) 20 re‑phrased language instructions, (v) cabinet translations up to 12 cm with rotations up to ±$\pi$/24, and (vi) random initial offsets on joints 0/1/2/4 of the end‑effector.

Among all splits, RL fine-tuning outperforms SFT on execution and semantic OOD tests, and performs comparably on vision, consistent with our findings on the pick-and-place task.

---

---

We deeply appreciate your thorough review and constructive feedback, which has enabled us to strengthen our work substantially. We are truly encouraged by your recognition of our study as “thorough”, as our primary goal was to provide the community with a systematic and reproducible empirical comparison of RL and SFT, rather than to propose a new state-of-the-art algorithm.

In response to your concerns, we have conducted a wide range of new experiments within the limited rebuttal period. We believe that these additional results further enhance the integrity of our study. We kindly ask you to reconsider your evaluation in light of our clarifications and the new experiments.

We sincerely appreciate the reviewer’s supportive comments on our contributions and the potential impact on the community. Below, we address your concerns clearly and thoroughly.

Q1: Clarification on $h^0$ and analysis of critic information

Thank you for pointing out the ambiguity regarding $h^0$. We will include an expanded and clarified explanation in the revised version.

• Detailed Explanation of $h^0$: We adopt the commonly used discrete action tokenization approach from prior work [1,2], where each action dimension is discretized into 256 bins and subsequently tokenized, projecting actions into a token space. Specifically, we denote the final-layer embedding corresponding to each action token as $h^i$, where the index $i$ refers to the $i$-th action dimension. Consequently, $h^0$ specifically represents the embedding associated with the first action token.

• Rationale for Selecting $h^0$ as Critic Input: In PPO, the critic is defined as the state-value function $V(s)$, which theoretically necessitates state-only information as input. In VLA models, observations encompass both visual inputs (images) and textual instructions, from which the VLM backbone extracts meaningful state representations. Then, a critical question arises: Why specifically choose $h^0$ over other embeddings for approximating $V(s)$, i.e., $V(s)\approx V(h^0)$?

  Our hypothesis is that the embedding serving as input to the critic should predominantly capture state information with minimal interference from action-related information. Empirically, $h^0$ sits precisely at the boundary between state and action embeddings in the auto-regressive VLA. To substantiate this choice, we have conducted experiments (Fig. 4b) demonstrating significant performance degradation when using embeddings that encapsulate more action-oriented information—such as $h^n$, corresponding to the last action token, or $h^{all}$, representing the concatenation of all action token embeddings. We attribute this performance drop to action embeddings introducing unstable, noisy signals during training, inherently conflicting with PPO's requirement that critic inputs strictly represent state information.

[1] OpenVLA: An Open-Source Vision-Language-Action Model. CoRL 2024.

[2] RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. CoRL 2023.

Q2: Why RL works for "grasp" skills but not general "perception"

We appreciate the reviewer’s thoughtful question. The distinction arises from which component of the VLA the generalization challenge primarily stresses. Our evaluation framework is explicitly designed to probe Vision, Semantics, and Execution generalization along separate axes, each of which targets different stages of the perception-to-action process.

In Semantics, the challenge lies in generalizing across object identities, instructions, and receptacle configurations. RL benefits from trial-and-error by refining its policy through interaction—e.g., learning how to retry or adapt grasping behaviors in response to failures. As shown in Fig. 7 and Fig. 9, this enables RL to outperform SFT in unseen object scenarios by learning robust strategies.

In contrast, Vision generalization introduces perturbations (e.g., novel textures, noise, unseen tables) that mostly challenge the perception module. While we do jointly train the vision encoder with RL, the signal from policy gradients remains sparse and task-driven, making it harder to recover from upstream perception errors such as mislocalization or occlusion. These errors degrade input quality before the agent can reason or act effectively. As a result, RL struggles to improve perception robustness beyond what is already achieved during pretraining, leading to comparable performance drops to SFT.

We will clarify this in the revision to emphasize that all three axes—Vision, Semantics, and Execution—are essential and complementary in evaluating VLA generalization.

Q3: Limited motion-planning-based data collection.

We appreciate the reviewer’s insightful comment regarding the use of motion-planner demonstrations. We acknowledge this as a potential limitation and have discussed it in the limitations section. In our experiments, we have tuned our motion planner to explore a larger space. Besides, we also tried to combine motion-planned data with demonstrations collected using a pretrained policy (e.g., Octo), as noted in Sec. 4.2 (lines 213–214). However, we found that the performance difference between motion-planned and Octo-collected data was minimal for SFT. Consequently, we scaled up motion-planning-based demonstrations (up to 64k) for our main SFT experiments.

Regarding the suggestion of injecting noise into the planner, we note that while simple noise can increase variability, it also risks degrading demonstration quality for the whole dataset. Designing noise that balances meaningful exploration with high-quality trajectories is non-trivial. Alternatively, more principled approaches—such as integrating neural motion planners [3] or further employing hierarchical pipelines like ManipGen [4]—could offer more robust data diversity. These are promising directions we plan to explore, but given the time constraints of the review process, we leave them as future work.

[3] Neural MP: A Generalist Neural Motion Planner. IROS 2025.

[4] Local Policies Enable Zero-shot Long-Horizon Manipulation. ICRA 2025.