STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantization

Thank you for your thoughtful review. We appreciate your recognition of our extensive experiments and the practical effectiveness of our approach. We address your questions below:

Q1：Have the authors considered comparing their method against QueST equipped with an offset prediction to isolate the contributions of the proposed combination more clearly?

We appreciate and follow your good advice by adding additional experiments that compare STAR against QueST across various configurations:

The results demonstrate that QueST with offset prediction (72.6%) significantly underperforms compared to our STAR method (94.3%)

We introduce offset head because our decoder is frozen during stage-1 training. This design means predicted actions can only reconstruct stage-0 actions, which undergo lossy compression through quantization, making fine-grained operations difficult. Therefore, we use an additional offset head to compensate for this gap.

In contrast, quest has already fine-tuned the decoder to learn fine-grained operations, which serves a similar purpose to our approach but through a different mechanism. Further adding an offset head does not yield significant gains and may instead conflict with the decoder's output.

To further verify the importance of learning fine-grained operations, we conducted an ablation study with QueST using a frozen decoder in stage-1. Performance dropped from 80.1% to 55.3%, consistent with our findings in Table 5 where removing the offset head significantly reduces our method's performance.

In summary, both approaches require mechanisms for learning fine-grained operations, but the superior performance of STAR comes from our novel contributions (RaRSQ and CST) rather than the offset head alone. The ablation studies in our original Table. 2 further support this conclusion by isolating the contributions of each proposed component.

Q2：It is currently unclear which component is primarily responsible for the performance gains. Could you further clarify to highlight the contribution of each component more clearly?

Our framework consists of two components: (1)rotation-augmented residual skill quantization (RaRSQ), which addresses codebook collapse in robotic skill learning, and (2)causal skill transformer (CST) , which captures causal relationship between skills through autoregressive prediction. We would like to further clarify the contribution of each component:

RaRSQ: Enhanced Skill Diversity and Representation

RaRSQ directly addresses the fundamental limitation of naive residual VQ-VAE - codebook collapse. As shown in Fig. 4, residual VQ-VAE utilizes only 43.8% codes, severely limiting the robot's ability to express diverse actions. In contrast, RaRSQ achieves 100% codebook utilization with balanced distribution.

This enhanced skill diversity translates to performance gains through precise skill decomposition. Complex manipulation tasks inherently require fine-grained action representation. With full codebook utilization, RaRSQ can distinguish between subtly different manipulation skills (e.g., picking different objects, precise positioning) that would otherwise be mapped to the same code in collapsed codebooks. This enhances the model's capability to recognize and execute task-specific manipulation patterns.

CST's Contribution to Skill Composition

CST's autoregressive design comes from analyzing the learned hierarchical dependencies between skills. The conditional probability analysis in Fig. 8 reveals causal relationships between first and second-level skills - some first-level codes show strong preferences for specific second-level codes. CST models the causal dependencies between skills, which is crucial for generating coherent action sequences. As show in Table. 2, when removing CST, we observe a significant performance drop, especially for LIBERO-Goal (-6.9%) and LIBERO-Long (-5.2%), indicating that modeling these dependencies is particularly important for complex manipulation tasks.

W1：While the integration of existing techniques is practically effective, the proposed method rely on adaptations and combinations of established ideas, which somewhat limits the theoretical novelty.

Our key contributions lie in designing a robot-specific residual VQ-VAE that effectively decomposes complex robot behaviors into discrete skills, addressing fundamental challenges in robot skill learning. RaRSQ directly addresses codebook collapse in robotic residual skill quantization, while CST explicitly models the causal dependencies between different skill abstraction levels, revealing structured relationships between coarse and fine-grained behaviors.

We appreciate your thoughtful comments and positive assessment of our work. Below we address the specific questions and concerns:

R1: A broader discussion on recent progress in robot learning—especially work on vision-language-action (VLA) transformers like π0—would provide valuable context regarding performance, efficiency, and scalability.

Indeed, we have already compared our method with several advanced VLA methods in our manuscript, including OpenVLA (7B) and Octo, as shown in Table. 1. Further, we appreciate and follow your good advice to include π0 in our evaluation. The results on LIBERO benchmark are summarized below:

Despite using significantly fewer parameters (200× smaller) and no pretraining, our approach achieves higher average performance (94.3% vs. 94.15%). Most notably, we observe the largest improvement on LIBERO-Long tasks (+3.3%), which aligns with our method's focus on addressing challenges in long-horizon tasks through effective skill composition.

VLA methods typically follow a pretraining-finetuning paradigm, with performance dependent on pretraining data and architecture (e.g., π0 > OpenVLA). Our work shows lightweight models trained from scratch can match or exceed VLA methods requiring extensive pretraining. Future work could integrate our approach with large language models to further advance robot learning capabilities.

Q1: What is the unique contribution of STAR compared to prior works in VQ-VAE, rotation-based augmentation, and VLA transformers?

We would like to further clarify that our key contributions lie in designing robot-specific residual VQ-VAE that effectively decomposes complex robot behaviors into discrete skills, addressing fundamental challenges in robot skill learning.

(1) Robot-Specific Solution to Codebook Collapse in Skill Learning

Rather than a straightforward application of rotation tricks to VQ-VAE, RaRSQ is specifically designed for robotic skill space with hierarchical structure. While residual VQ-VAE offers larger representation space, this makes codebook collapse more problematic. Our integration of rotation-based gradients within the residual framework preserves geometric relationships throughout hierarchical action quantization, effectively capturing both coarse primitives (move, pick) and fine-grained adjustments required for complex manipulation.

(2) Explicit Modeling of Hierarchical Skill Dependencies

Our CST differs fundamentally from previous VLA transformers by explicitly modeling conditional dependencies between different skill abstraction levels. Unlike approaches that predict actions directly or treat skills independently, CST captures structured relationships between coarse and fine-grained behaviors. As shown in Fig. 7, it reveals distinct skill dependency patterns that validate this approach. For example, given a first-level code, we found strong preference for specific second-level codes, showing how coarse skills constrain fine-grained behaviors in ways prior approaches cannot capture.

As shown in Table. 2, removing the autoregressive component significantly reduces performance, , especially in LIBERO-Goal (-6.9%) and LIBERO-Long (-5.2%). In addition, removing both components creates a 5.8% drop over all tasks, exceeding the sum of individual effects, demonstrating the synergistic relationship between skill representation and composition.

Q2: Can you confirm if the use of r_d in both eq(8) and eq(11) is correct, or if it is a typographical error?

Thanks for pointing out the typo. $r_d$ in eq(11) should be $\hat{r_d}$, and we will correct it in the final revision.

Q3: Could you provide more details or video on the real-world robot experiments as well as completion time or speed?

We have recorded comprehensive videos showing the complete execution sequences for both tasks (drawer manipulation and sequential object placement). Since ICML rebuttal guidelines explicitly support anonymous links for supplementary figures and tables but don't specifically videos, complete videos will be included on our future project page.

Regarding completion time, STAR demonstrates favorable completion times, averaging 29.6 seconds for the sequential object placement task and 37.7 seconds for the drawer manipulation task. This efficiency is attributable to our hierarchical skill abstraction approach, where a single inference step produces an action chunk (consisting of eight atomic actions in our implementation). This reduces the number of required inference steps and consequently decreases overall execution time. We will incorporate these timing metrics in the final version to provide a more comprehensive evaluation.

We sincerely thank Reviewer G1Cu for the thorough and positive assessment of our work. We appreciate your recognition of our key contributions in improving codebook utilization through rotation-augmented residual skill quantization and implementing autoregressive decoding for effective skill composition.

If you have any further questions, we would be more than happy to address them.