Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models

Thanks for your review and the additional references! Please find below our answers.

The problem definition is not very clear, making it difficult to understand at the beginning.

We consider a setting where a model can be pre-trained from a dataset of unlabeled observation-only trajectories and online interaction with the environment. At test time, the resulting model should be able to solve different types of tasks, including trajectories tracking, goal reaching, and reward optimization. We will further clarify the setting informally in the introduction and formally in section 2.

The theoretical explanations about FB representation might be a little complicated ... some explicit examples or pictures may help

Due to space constraints, in the main paper we have focused on the core aspects of the FB model and its losses to allow the reader to understand the key differences we introduced in the FB-CPR model. We will expand on more algorithmic aspects and theoretical properties in the supplementary material to make our contribution more self-contained including illustrations of the inference process.

It is unclear how the advantages and differences of this method compare to previous approaches, such as AMP[1] and ASE[2], as well as to recent work (like Omnigrasp[3], MaskedMimic[4]).

AMP and ASE share a similar intuition that policy learning should be regularized by some additional imitation learning objectives. While AMP is designed for single-task problems, ASE aims at pre-training a behavioral foundation model. The main differences between ASE and FB-CPR are: 1) ASE employs an unconditional discriminator which encourages policies to generically cover the same states as in the behavior dataset. This does not guarantee that the resulting model can actually reproduce any of the motions in the dataset. On the other hand, the conditional-policy discriminator of FB-CPR encourages learning policies that can reproduce fragments of the trajectories shown in the dataset, which makes FB-CPR better at tracking. 2) ASE relies on a DIYAN objective to enforce diversity across policies, whereas FB-CPR leverages an FB loss component which favors policies that are optimal for some reward function. This leads FB-CPR to express policies that are better at reward optimization than ASE. Please refer to Table 1 for an extensive comparison. Omnigrasp is a version of PHC specifically designed for object manipulation and it is focused on tracking tasks. We report a comparison of FB-CPR with PHC in Table 1. Finally, MaskedMimic, which was released only a week before the submission, relies on a complex pipeline where first an imitation policy is learned (using a carefully crafted reward) and then it is distilled in a masked version of the policy to support different downstream use cases. Unlike FB-CPR the resulting model does not support reward inference and it rather relies on hand-defined finite-state automata (``goal-engineering’’) to solve more complex downstream tasks.

I want to know if it is possible to deploy the method on real robots, just like RobotMDM[5] (I feel that this method is difficult to have this scalability).

FB-CPR does assume access to the environment through direct online interaction. While this is not desirable in real robotic applications, we could follow the standard sim2real protocol by first training FB-CPR in simulation and then deploy it and fine-tune it on an actual robot (the same approach is used in RobotMDM). The other assumption we have is that behavior data are expressed in the same embodiment as the agent we are training. In a robotic application, this could be obtained by data collection from the robot itself and/or from retargeting e.g., motion capture datasets. Regarding scaling, we do not anticipate any specific challenge since FB-CPR is already trained on a humanoid with 23 joints, whereas in RobotMDM a bipedal robot with 20 degrees of freedom is considered.

While the paper focuses on motion capture datasets, how would FB-CPR perform when trained on more diverse or noisy datasets, such as uncurated video footage?

In the short time of the rebuttal, unfortunately we could not run additional experiments from video datasets. Nonetheless, we would like to point out that the AMASS dataset already contains motions with noise, recording and reconstruction artifacts, and in general they may not be realizable in the physics-based simulation we consider (i.e., there may not be any sequence of actions able to reproduce the same transitions). This is already a significant departure from the protocols often used in RL literature, where demonstrations datasets are generated by rolling out policies in the actual environment without any additional noise, thus avoiding any non-realizability issue.

Thanks for your review! We addressed your questions in the general answer and in the following replies.

The majority of the evaluations are conducted only on the humanoid environment.

We have now included experiments in the AntMaze domain from the recent OGBench benchmark. Please refer to the general answer and the revised paper for more details.

Although more “human like” behavior (more like the training dataset) might be desirable in a humanoid environment, could the regularization negatively affect performance where the majority of the data is very suboptimal?

In general, the “behavior dataset” is intended to restrict the scope of unsupervised RL and focus it on tasks/policies that are somehow related to the data. While FB-CPR is indeed trying to learn policies that can reproduce segments of the demonstrations, the FB part of the loss is also pushing towards learning optimal policies for rewards in the span of the representation B. In the humanoid case, most motions are not generated by a stationary Markov policy optimizing a reward function. As such, most of them are heavily suboptimal for the reward functions that we consider in the reward-based evaluation. Nonetheless, FB-CPR manages to learn policies achieving satisfactory performance in most of the tasks, while retaining the human-like nature of the demonstrations.

Furthermore, in the new AntMaze domain and in the original ablations performed in the bipedal walker domain (In App. E) we do not target any “qualitative” bias and the behavior datasets were only used to ground unsupervised RL. While the demonstrated behaviors may be optimal for some specific tasks, they are suboptimal for the reward functions used at test time. Also in this case, the regularization is effective in skewing the learning towards policies that are “similar” to the demonstrations, while achieving good performance in the reward-based tasks.

It feels like something like METRA [1], which is explicitly trained to span a diverse set of behaviors, is also a relevant baseline.

Thanks for the suggestion! We have implemented METRA and some related variants. Please refer to the general rebuttal and App. H in the revised paper for further details.

On the humanoid experiments, why aren’t there more direct comparisons to FB without the proposed regularization?

Please refer to the general answer and the additional experiments in App. D.2 in the revised paper. Unfortunately basic FB trained online achieves very poor performance as it does not collect useful samples and in turns it does not learn any effective behavior.

How does the proposed regularization affect the nature of the embedding space? Could a nicely regularized latent space enable something like interpolations between skills etc.?

Thanks for the question and the suggestion! We have spent some time digging more into the structure of the embedding space and we have now convincing illustrations on how the latent space effectively clusters motions together while aligning them with reward embeddings. Furthermore, preliminary tests on task interpolations shows that the policy embedding varies the behavior smoothly and it is capable of composing tasks with different objectives (e.g., "spin" and "move" produces "spinning and moving", while "move" and "raise arm" produces "move while raising arm"). Please refer to revised paper and supplementary material, and the general answer for more details.

Thanks for your review! We have submitted a revised version with additional experiments (see general answer) and we address your questions below.

One weakness is its limited novelty.

While we agree with the reviewer that combining a policy optimization loss with some imitation regularization is a fairly common principle in the RL literature, its instantiation into unsupervised RL is non-trivial and the specific algorithmic solution we propose with FB-CPR is novel and, more importantly, it is crucial to obtain the significant improvements reported in the paper compared to a wide range of baselines.

More in detail, the FB method requires access to dataset with good coverage, it is trained fully offline, and it optimizes policies in a completely unsupervised way. While working well in small domains, it does not scale to more challenging problems, such as humanoid. FB-CPR resolves the limitations of FB through the imitation learning regularization as well as an online training approach. Compared to other regularized unsupervised RL approaches (e.g., ASE, CALM), FB-CPR leverages the FB components for trajectory encoding (unlike CALM that requires training a dedicated encoder) and to preserve policy optimality (unlike ASE, which builds on diversity principles). As demonstrated in Table 1, these differences are crucial to make FB-CPR perform significantly better than other baselines. FRE (similar to HILP) is an offline unsupervised RL algorithm and it proceeds through a two-step process to first learn a task encoding and then optimize policies accordingly using a standard offline regularized RL algorithm (IQL), which requires access to actions in the offline data. On the contrary, FB-CPR has access to observation-only datasets and it trains representations and policies end-to-end in an online fashion and the regularization is based on a conditional discriminator, with a significantly different objective than IQL. Finally, we have performed ablations and compared FB-CPR with several different variations of the regularization: 1) FB-AW trained offline on the action-labeled AMASS dataset (Fig. 4-bottom right); 2) FB-CPR trained online with BC regularization using the action-labeled AMASS dataset (Fig. 6-bottom right); 3) FB-CPR but with an unconditional discriminator (Fig.4-top left). In all cases, the specific combination of FB training and conditional-policy regularization is the crucial ingredient to achieve the best performance across all problems we considered.

I'm curious why FB-CPR is only shown on Humanoid control.

We primarily focused on the humanoid problem due to its dynamical complexity, high dimensionality, availability of human data, and the possibility of defining a large set of “natural” tasks. Unfortunately no other existing RL benchmark have all these properties at the same time. Nonetheless, we have included in the revised version of the paper additional experiments in the AntMaze domain recently introduced in the OGBench benchmark, which provides a meaningful dataset of short trajectories designed to test stitching capabilities of unsupervised RL algorithm as well as a few downstream tasks to evaluate generalization performance. Please refer to the general answer for more details.

Thanks for your review! We addressed your questions in the general answer and in the following replies.

One core assumption of the work is that one can reparameterize the policy dependence with the introduction of a policy embedding z (Eqn 4). I wonder how grounded is this, especially when z has to live in a continuous space.

In our experiments, the policy $pi_z(s)$ is a z-conditioned network with two initial “embedding layers”, one processing (s,z), and the other processing the state s alone. The second embedding layer has half the hidden units of the first layer, and their outputs are concatenated and fed into the main MLP. In FB-CPR the overall network is then optimized through the loss in (11), which allows us to learn a continuously parameterized policy that can generalize across states as well as the embeddings z.

What's the dimension of z? Usually, a policy has to be represented by a neural network. I wonder whether a compact latent variable z is expressive enough to represent policies that have combinatorial complexity. In that case, you must be sacrificing something. Could you provide more intuitions and conduct more ablations on the dimension of z?

The state and action space of the humanoid agent are 358-dimensional and 69-dimensional respectively, while in our experiments we use a 256-dimensional Z space, which clearly does not allow expressing the whole combinatorial set of policies. Nonetheless, our model exploits two inductive biases that allow using its capacity to express policies that more relevant to the problem: 1) the conditional-policy regularization helps focusing the model on the “manifold” of human-like behaviors that is much smaller than all possible policies; 2) the low-rank decomposition in the FB models favors representations that capture variables with slower dynamics, hence inducing tasks whose optimal policies tend to generate more steady behaviors. Please refer to App. D.2 Fig. 7 for an ablation on the dimension of the embedding.

I wonder whether TD3 is a competitive baseline at all for "Naturalness"

We prioritized TD3 over other models because the human evaluation was intended to primarily investigate whether the performance gap between FB-CPR and TD3 in reward-based and goal-based tasks could be partially explained by TD3 exploiting the physics of the humanoid model to optimize performance at the cost of “human-like” behaviors. While it is difficult to design a top-line with the best performance under a “human-like” constraint, we believe this evaluation provides a first qualitative assessment that FB-CPR may trade off performance and qualitative behavior and it is able to carry over the human-like regularization across reward-based and goal-based tasks.

It seems that most of these works are evaluated on relative toy datasets only e.g. maze. Therefore, it's a big step for the authors to make claims about an application like humanoid control while skipping evidence of FB & state measure methods being general enough for RL benchmarks.

While still relatively recent, the FB model is covered in a variety of previous works (e.g., [Touati and Ollivier, 2021]) and it has been tested in several RL benchmarks including discrete and continuous mazes, FetchReach, Ms.Pacman and most of the environments in the URLB benchmark, and it is included as baseline in other unsupervised RL works (e.g., [Park et al., 2024b]). In particular, [Touati et al., 2023], [Pirotta et al, 2024] performed an extensive comparison between FB-based models and several reward-based and imitation-learning baselines, showing its effectiveness in mid-scale problems whenever good coverage offline datasets are provided. Due to space constraints, in the main paper we have focused on the core aspects of the FB model and its losses to allow the reader to understand the key differences we introduced in the FB-CPR model. We will expand on more algorithmic aspects and theoretical properties in the supplementary material to make our contribution more self-contained.

Please refer to the general answer for additional experiments in AntMaze (and walker). In general, we found that many existing benchmarks for unsupervised RL are defined on environment with simple low-dimensional dynamics, or do not have datasets of ``meaningful'' behaviors, or do not support more than a handful of hand-picked tasks. We believe that the humanoid benchmark we introduced in the paper provides a challenging and exhaustive evaluation that could help to advance research in this domain.

Figure 2 and Algorithm 1

Thanks for the suggestions! We will update Fig.2 clarifying the FB components and illustrating the inference part. We will also add a compact version of Alg.1 if space permits it. Furthermore, we plan to release the training code together with the trained models and the Humanoid environment and benchmark in a few weeks.