From Experts to a Generalist: Toward General Whole-Body Control for Humanoid Robots

We thank the reviewer for their valuable suggestions. In our response, we primarily address potential misunderstandings, as well as the weaknesses and questions raised in the review.

Since the method is modified from the prior method ASAP, it should explicitly use ASAP as the experimental baseline for comparative evaluation. Although the paper compares some algorithms for migrating from H1 to G1 robots, I think a direct comparison with methods primarily designed for G1 robots would be more appropriate.

We first address the fundamental differences between our method and the ASAP approach. ASAP is inherently limited to single-motion settings, as it represents task information solely through the motion phase in the observations. This makes direct comparison with our method impossible.

However, to validate the distinction between our method and ASAP, we conduct experiments in which the single delta action model, trained on the entire real dataset, is used to finetune the initial general policy, as described in the paragraph starting at L. 302. As discussed at L. 309, the result shows that this single delta model underperforms in diverse motion settings, supporting our claim that ASAP-style delta modeling does not generalize well across heterogeneous motion classes.

1. The merged motion demo appears to exhibit lower-body instability in real-world deployment. Without the support of suspension ropes, it remains uncertain whether the system can maintain stable standing.
2. The lower body in the merged motion demo appears somewhat unstable. What could be causing this behavior?

The relatively intense lower-body movements observed in the merged motion are primarily due to the task’s high difficulty and large motion range. To maintain whole-body balance, the robot naturally exhibits more compensatory behaviors, especially in complex motion sequences involving boxing. In contrast, for simpler categories such as stand-up, the policy produces highly stable motions, as demonstrated in the two accompanying videos.

Furthermore, across all videos, the rope attached to the robot remains mostly slack, with the rear rope consistently loose. Even when the cabinet moves and the rope trails behind, the resulting force is predominantly horizontal and does not contribute any meaningful support to the robot’s standing behavior. Given that only a single robot was available for the demonstration, we have to employ a rope as a safety measure to protect the hardware.

While the proposed method categorizes motions into six classes, how should the system accommodate new action types (e.g., squat) – through additional class expansion or alternative architectural modifications?

AMASS already covers a highly diverse range of human motions, and our current six-category division captures a broad spectrum of kinematic behaviors. Importantly, our clustering is based on kinematic characteristics rather than only semantic labels, meaning that any new motion can, in principle, be assigned to one of the existing clusters according to its movement profile—for example, squatting naturally falls under the “stand-low” category in our setting.

Moreover, our expert-to-generalist framework is inherently modular and extensible. The MoE-style structure allows new motion types to be incorporated simply by training additional expert policies tailored to additional category, which can then be distilled into the unified generalist policy. Since only the final distilled model is used for deployment, expert architectures can even differ in network architecture without affecting integration.

I observe that ASAP has a more natural jumping effect in a real-world experiment. Are there some examples of comparisons that BB can accomplish but ASAP can't? I think it's important to evaluate the contribution of the BB method.

As noted in our previous response regarding the use of ASAP as a baseline, the task settings of ASAP and our method are fundamentally different. BumbleBee is explicitly designed for generalizing across a diverse set of motions, whereas ASAP is tailored for single-motion reinforcement learning. Its policy architecture does not support cross-task generalization, which would require training a separate model for each of the 10,000+ motion sequences in AMASS—an approach that is both computationally prohibitive and impractical.

In contrast, our method is inherently suited for multi-task learning. By clustering motion types based on kinematic characteristics and training expert policies accordingly, BumbleBee can generalize to a wide range of tasks. Moreover, since BumbleBee is trained on over 8,000 motion clips derived from real human data, it demonstrates robust generalization capabilities even without explicit supervision for each individual motion.

The experiment does not account for the impact of random seeds on the experimental results.

We observe that the variance of policies trained in IsaacGym is very small, consistent with prior works like ASAP and HOVER. This is likely due to the large batch size (envs=4096) used during training, which minimizes randomness across different seeds, resulting in more stable performance. Importantly, the small variance does not cause any overlap and does not affect the results. To keep the main text concise, we have not included full statistical tables, but the complete statistical results are provided in the table below.

And we will add it into the final version.

We appreciate the reviewer’s feedback and have focused our response on addressing the aspects of the paper that may require improvement.

A thing that might be interesting to explore i the future is how the quality of the method changes with the number of clusters, and maybe even more interesting, how the methods scales for more classes of substantially different movements.

Does it always make sense to take semantic similarity into account? It appears that it might be somewhat dependent on the quality of the semantic labeling and the specificity of semantic labeling. For example, 'turning' might include a whole range of movements that are much more variable than different types of walking (slow fast etc.) Can it be guessed how the method would, extend for such more heterogeneous motion classes ?

As illustrated in Fig. 3, we selected a moderately balanced number of clusters that offers a practical trade-off between task granularity and implementation complexity. In essence, clustering is used to simplify the learning problem: with more clusters, each expert is trained on a narrower and easier subtask, which typically improves expert performance. However, increasing the number of clusters also significantly raises the overall workload and complicates the distillation process. Given this trade-off, we opted for a reasonable balance in our current setup. We acknowledge the importance of this parameter and will make an effort to investigate its impact in the final version.

We share the same concern regarding semantic similarity, as language-based labels can suffer from quality issues, ambiguity, and redundancy. For instance, the label "playing basketball" may correspond to a range of motions, such as dribbling while moving or standing still to shoot, which are kinematically distinct despite sharing the same semantic tag. To address this, we do not rely directly on semantic labels for clustering. Instead, we align motion and text into a shared latent space and perform clustering based on this representation. This allows for a more meaningful categorization that captures both fine-grained motion differences and underlying semantic relationships, as illustrated in Table 2.

• What is the meaning of k(s) in L. 191. Is k an index?

Here, k refers to the index of the expert corresponding to the motion that the given state belongs to. Thank you for pointing out and we will revise it accordingly.

• L. 319: There is the class 'RUN' mentioned that seems not to appear anywhere before. Is this fast walking?

We appreciate your observation. In this case, the label "run" was indeed intended to represent a fast walking motion. We will revise it in the final version.

To reviewer 4Wt3

We sincerely thank Reviewer 4Wt3 for the insightful comments. We mainly clarify the points that may have caused confusion and address the weaknesses.

Clarification for design decisions

How much of the benefit of BB comes simply from improved NN architectures/other hyperparameter choices, vs. the clustering, expert, and distillation steps emphasized in the paper? It would be good to include the full set of metrics for “General Init” in Table 1. If most of the benefit is coming from differences other than the clustering approach, these differences should be described in more detail.

Thank you for the thoughtful question. To ensure fairness, we kept the MLP architecture consistent across all models, including ablations and baselines. The only exception is BUMBLEBEE, where we employed a Transformer-based architecture specifically for the purpose of distilling multiple experts more effectively. And thank you for your remind, we will include the "General Init" into Table 1. We also conduct an additional experiment that use the same MLP as backbone of BB and get the success rate result of 89.56% in IsaacGym and 61.04% in MuJoCo. The MuJoCo result is slightly lower than BB, but it is still significantly higher than the baseline, which proves that while the architecture does have an impact, the main influence comes from the expert and clustering strategies.

Statistical tests

We observe that the variance of policies trained in IsaacGym is very small, consistent with prior works like ASAP and HOVER. This is likely due to the large batch size (envs=4096) used during training, which minimizes randomness across different seeds, resulting in more stable performance. Importantly, the small variance does not cause any overlap and does not affect the results. To keep the main text concise, we have not included full statistical tables, but the complete statistical results are provided in the table below.

And we will add it into the final version.

Parameter count

While the paper includes ablations showing that the use of multiple experts improves performance over the generalist policy (e.g. Table 3), it’s still not clear to me why this is the case. Do the generalist and 6 experts all use the same model architecture described in Appendix C? If so, maybe the greater total capacity of the experts allows better performance, which can then be distilled. What if, instead of 6 separate experts, you trained a single “General Init” using a model with 6x the total parameter count, then distilled this down to a single model of the original size? This would eliminate the need for the clustering models and significantly simplify the full method; unclear based on the paper whether we should expect this to work?

As requested, we conducted an additional experiment by increasing the parameter size by 6× while keeping all other settings identical to the "General init" configuration. The success rates were 85.17% in IsaacGym and 37.36% in MuJoCo, showing minimal performance improvement over the original model. This further confirms that simply increasing the model size has limited impact on performance.

Detailed description of the metric

Some unclear environment descriptions: what is “Success Rate”? The paper says “Among all metrics, SR is the most critical, as it reflects the overall viability and stability of the policy” and “Detailed definitions of these metrics can be found in Appendix A.”, but I’m not able to find these definitions in the provided appendix A.

Thank you for the feedback. In our setting, "Success Rate" (SR), shown in line 813 of appendix, refers to the policy successfully completing the task without failure. More specifically, a trial is considered successful only if the agent maintains balance (i.e., does not fall) and closely follows the reference motion. A policy that remains stable but deviates significantly from the target motion would still be considered a failure. We will include SR and the two additional metrics in Appendix A.

The Mean Per Joint Position Error (MPJPE):

$$\text{MPJPE} = \frac{1}{N} \sum_{i=1}^{N} \left\| \hat{\mathbf{J}}_i - \mathbf{J}_i \right\|_2$$

• $N$: Number of joints
• $\hat{\mathbf{J}}_i \in \mathbb{R}$: Target joint position of the $i$-th joint
• $\mathbf{J}_i \in \mathbb{R}$: Joint position of the $i$-th joint.
• Unit: rad

The Mean Per Keypoint Position Error (MPKPE) :

$$\text{MPKPE} = \frac{1}{N} \sum_{i=1}^{N} \left\| \hat{\mathbf{K}}_i - \mathbf{K}_i \right\|_2$$

• $N$: Number of keypoints
• $\hat{\mathbf{K}}_i \in \mathbb{R}^3$: Target position of the $i$-th keypoint
• $\mathbf{K}_i \in \mathbb{R}^3$: Position of the $i$-th keypoint
• Unit: mm

To reviewer sTVB

Thanks for your detailed reviews.

Hardware experiments

• As alluded to above, it would be rather insightful if the authors could provide some quantitative evaluation metrics for the hardware experiments. I understand that computing the same performance metrics as in simulation may be difficult, but on the other hand it is also difficult to evaluate the effectiveness of the expert refinement + final generalist performance without any metrics or baselines / comparisons.

We agree with that it is important to provide quantitative evaluation metrics for the hardware experiments.

Also, it is challenging to provide precise quantitative comparisons for real-world deployment, primarily due to the lack of publicly available deployment code for baseline methods, which makes fair sim-to-real comparisons difficult. To address this, we present extensive videos demonstrating the capabilities of our final policy and comparisons with and without delta fine-tuning in each cluster. Additionally, we report results for our final general policy on 100 randomly sampled motions from the AMASS dataset, achieving a success count of 59 and an MPJPE of 0.2896 — slightly lower than its performance in MuJoCo, but still demonstrating strong real-world generalization.

Gap between IsaacGym and MuJoCo

• I am somewhat surprised to see such a large gap in performance between methods in MuJoCo when numbers are rather close in IsaacGym. The authors argue that this is because MuJoCo is a more realistic simulator, but I find that to be a rather handwavy explanation for such a dramatic difference in results. Could the authors please elaborate on why there might be such a discrepancy between the two on a more technical level, and e.g. include some videos of each method evaluated in MuJoCo? Do baselines tend to overfit to the IsaacGym simulator? If so, could that possibly be alleviated by improving policy robustness with domain randomization or added control noise?

We appreciate the reviewer’s thoughtful question regarding the performance discrepancy between MuJoCo and IsaacGym. We believe several factors contribute to this difference.

1. The physics engine. The discrepancy in results between MuJoCo and IsaacGym can be partially attributed to differences in their physical simulation fidelity. Specifically, MuJoCo provides more accurate and tunable contact modeling, which are often approximated more coarsely in IsaacGym. Additionally, options such as collapse_fixed_joints=True are often used to reduce model complexity and improve simulation speed. While this can be beneficial for large-scale training, it effectively removes certain physical structures that are present in the original URDF models. Furthermore, joint-level control in IsaacGym may suffer from limited temporal precision, making it challenging to accurately regulate the duration and timing of torque application. In contrast, MuJoCo provides fine-grained control over both the timing and magnitude of applied torques, which is essential for precise motor behaviors. Finally, MuJoCo offers detailed control over joint friction and damping. All above can significantly impact performance in tasks especially involving locomotion or contact-rich behavior that account for a significant portion of the AMASS dataset.

2. Observation space. We hypothesize that different observation space designs exhibit varying levels of robustness in domain transfer scenarios. Compared to the local-coordinate-based observation space used in our method and ExBody2, world-frame-based observations—such as absolute keypoints positions—are significantly more fragile. This is because even minor perturbations or mismatches in the environment (e.g., initial pose differences) are directly reflected in the observations, amplifying their impact on policy performance. In our experiments, MuJoCo initializes the robot to a default pose that approximates real-world joint configurations, while IsaacGym resets the robot based on the reference motion. This discrepancy introduces an initial state mismatch between the two simulators. As a result, policies relying on world-frame position, such as absolute keypoints positions, suffer greater performance degradation under domain transfer, as also observed in ExBody2 for contrast. This issue becomes even more critical in real-world deployment, where precise resetting to a reference pose is generally infeasible.

3. Overfitting in IsaacGym. We acknowledge that overfitting during training in IsaacGym is a potential concern. In our experiments, we applied observation noise and domain randomization to the same extent as in ASAP. However, excessively large noise or overly aggressive domain randomization can significantly degrade training performance, and identifying the optimal trade-off remains a well-known challenge across all robot control algorithms. ASAP itself has been widely recognized as an effective approach for sim-to-real transfer. Building on this foundation, we extend the framework by introducing motion category diversity, enabling the learned general tracking policy to achieve better domain transferability.

We apologize for not being able to include additional figures at the rebuttal stage. However, as shown in Fig. 5, similar issues are observed in the MuJoCo environment as well — the policy often exhibits noticeable foot instability, struggles to accurately follow the reference motions, and in some cases, crashes early due to significant initial state mismatch. These issues may be attributed to the factors discussed above.

Number of clusters

• Given that clustering places a key role in this work, I expected to see an ablation on the number of clusters. I understand that this could potentially be quite time consuming if it includes real world policy refinement, so perhaps a simpler ablation could be set up purely in simulation. Even so, this might not be feasible within the strict time frame of a rebuttal. It would be helpful to include some discussion on the (expected) effect of the number of clusters on the overall approach, ideally backed by empirical evidence for e.g. the camera-ready version.

Thank you for the suggestion. We agree that exploring different numbers of clusters is valuable, but it also involves substantial additional effort. As illustrated in Fig. 3, we selected a moderately balanced number of clusters that offers a practical trade-off between task granularity and implementation complexity. In essence, clustering is used to simplify the learning problem: with more clusters, each expert is trained on a narrower and easier subtask, which typically improves expert performance. However, increasing the number of clusters also significantly raises the overall workload and complicates the distillation process. Given this trade-off, we opted for a reasonable balance in our current setup. That said, we acknowledge the importance of this parameter and will make an effort to investigate its impact in the final version.

Supplemental videos

We apologize for the inconvenience. You may try accessing the alternative link provided in the anonymous link, or download the video directly. This is currently the best method we can offer to share the video anonymously.

Suggestions for the paper

I like the general idea behind Figure 4; it neatly summarizes the empirical expert vs. generalist results. However, the different methods are currently difficult to visually distinguish due to the choice of colors. I would recommend refining this figure to better separate the different methods visually.

1. Thank you for pointing this out. The data presented in Fig. 4 is indeed information-dense, and the color scheme may be too similar to distinguish clearly. We will revise the figure in the final version to improve clarity.

The discussion of limitations is rather minimal and merely points out that additional sensors are likely to improve results. I strongly suggest that the authors revise their limitation section to discuss a few key limitations pertaining to the proposed method itself, e.g. the way that clustering is currently performed, or the scalability of the generalist-to-expert-to-generalist training pipeline.

2. We appreciate your valuable feedback. In the revised version, we add some of the method's limitations pertaining to the proposed method itself, including the two issues you mentioned, the overall methodological complexity, and the significant amount of real-world iteration that may be required for practical deployment.