Humanoid Locomotion as Next Token Prediction

Thank you for your comments and suggestions! Please find our responses below:

As shown in Figure 8, the prediction error is correlated to the tracking error. I am curious about how much data the authors need to make these two errors correlated, since from my knowledge, the real world is much more complex and it is usually hard to have such a correlation,

We use a total of 30k trajectories for training. Since we evaluate the prediction error on a held out validation set we think that the errors may be correlated in a lower data regime as well (less training data may lead to higher prediction error and higher tracking error as well). Please let us know if this answers your question.

Recently, there have been numerous impressive advancements in humanoid robotics. While these developments are exciting, it is unfortunate that most of these works do not open-source their code. I understand the reasons behind keeping the code proprietary, but open-sourcing it would provide significant benefits to the broader community. I believe there are a lot of engineering details behind the simple framework, both in the algorithm and in the real world deployment, which however might not be shown in the paper.

Thank you for the suggestion. We commit to open-source the code, models, and data used in this study to facilitate future research in this area.

Thank you for your comments and suggestions! Please find our responses below:

Additional details on some experiments.

Our goal in Figure 5 is to give a qualitative sense and we use one trial for each command. The following table shows a quantitative comparison between our model, RL, and MPC models. We report the mean velocity tracking error of different models under different yaw commands (250 different yaw values sampled in total).

We can see that MPC has the highest tracking error among the three methods while our model has the better performance across all the commands. We will update the comparisons to MPC, as well as the hyperparameters, in the next version of our paper.

Ablation on quantization.

Our model uses MSE loss on continuous observations and actions. We compare it to using cross entropy (CE) loss on quantized observations and actions. We use uniform binning for quantization, i.e., for each dimension of the observations and actions, we split the value of that dimension into N uniform bins and the observation/action prediction is formulated as a classification problem of predicting which bin the value of that dimension falls into. We use N=100 for all dimensions. The results are shown below:

We can see that using MSE loss has a better performance. Note that here we only use a vanilla quantization method of uniform binning and using more advanced approaches such as non-uniform binning or vector quantization might provide better performance.

Ablation on mask token.

Our model uses mask token to replace the missing modalities in model-based, MoCap, and human video data. We compare using a leanable mask token with using a constant vector (e.g., zero vector) to fill in the missing modalities. Results are shown below:

We can see that using mask token is slightly better than using a constant vector.

Clarification on modality-aligned prediction.

Given a sequence of interleaved observation and action tokens [$o_1$, $a_1$, …, $o_{t-1}$, $a_{t-1}$, $o_t$, $a_t$, …], the naive next token prediction will predict $o_t$ from $a_{t-1}$ and then $a_t$ from $o_t$. For modality-aligned prediction, we predict $o_t$ from $o_{t-1}$ and $a_t$ from $a_{t-1}$, i.e., we always predict the token from the previous token that belongs to the same modality.

Ablation on noisy pre-training vs. joint training.

We compare two ways of training our model: 1) first pre-training on noisy observation prediction on MPC, MoCap, and human video data, and then fine-tuning on action prediction on RL data. 2) training the model jointly on observation and action prediction on all data. Results are shown below:

We can see that joint training is slightly better. The results are also reported in Table 1c in the paper.

Details on the modalities used.

There are in total two modalities: observation and action. The observation is a 102-dim vector which is the concatenation of the joint positions of each joint/motor, the velocities of each joint, the velocity of the body, the gravity, the command, etc. The action is a 36-dim vector which contains the commands it will send to each motor for the next step. During inference, at each step the model predicts both the observation and action vector for this step. Assuming the input is the history of observations and actions [$o_1$, $a_1$, …, $o_{t-1}$, $a_{t-1}$], the model predicts $o_t$ and $a_t$ at the t-th step. The causal mask is constructed such as any modality in each step (e.g., $o_{t-1}$) will only see other modalities in the same step ($a_{t-1}$) as well as all the modalities in the previous steps (e.g., $o_{t-2}$ and $a_{t-2}$).

The prediction error for stage training (Table 1c).

For stage training, the model is pre-trained to predict observations in the first stage, and then fine-tuned with action prediction in the second stage. Since the prediction error counts in both observation prediction and action prediction, and the model is not supervised with observation prediction in the second stage, it is probably not fair to compare with the model that is trained jointly on observation and action prediction, which is why we didn’t report the number. For reference, the prediction error for stage pretraining is 42.22 which is way higher than the prediction error of joint training (0.88).

Was predicting both observation and action at the same time tried?

Our model predicts both observation and action at the same time. Please also see the clarification on modality-aligned prediction above.

Thank you for your response. We are glad to hear that we addressed all of your initial comments. We would like to offer some clarifications regarding the related work discussion.

We believe that reviewer FS4P’s suggestion about an additional reference in robotic manipulation was a minor point, which we agreed to include. Reviewer FS4P acknowledged that this concern was fully addressed in their response and also noted the novelty of our work in their review.

We want to emphasize that we do not claim "Autoregressive Pre-training for Robotics" as our primary contribution. In fact, we already have the "Transformers for Robotics" section in the related work that discusses a number of papers on autoregressive pre-training in robotics. We are happy to expand this section further.

The novelty of our work lies in casting real-world humanoid locomotion as a joint distribution learning problem with transformers. We further show that this formulation enables training with noisy or missing data (e.g., derived from videos) and achieves excellent results in the real world.

While our work can have excellent impact on multiple areas, the paper and title focus on humanoid locomotion. We chose this problem as a test bed because it is particularly challenging due to its highly dynamic, complex nature. The success of our method in this domain is particularly noteworthy (few prior approaches used learning and the ones that did rely on RL).

Given these clarifications and the original review, we believe the original score more accurately reflects our contribution and that it would be unfair to reduce it. We respectfully ask you to reconsider maintaining your initial score.

Thank you for your response. We appreciate your acknowledgement of our contributions and results. We would like to clarify a few points below:

• Autoregressive pre-training is not a primary contribution of our work. While our approach can be used with pre-training, our method does not involve pre-training. We are "just" training a model from scratch and deploying it zero shot, without multi-stage training or fine-tuning.

• We are more than willing to expand the “Transformers for Robotics” section with work on autoregressive pre-training in robotics to provide a more comprehensive context.

• Our primary contribution lies in formulating real-world humanoid locomotion as a joint distribution learning problem, enabling training with noisy or missing data and achieving very strong results in a particularly challenging domain.

Thank you for your consideration.

Thank you for your comments and suggestions! Please find our responses below:

Claiming that training with video might be overclaiming; in fact, it only uses human poses from the video, which is quite different from training with video.

We pre-process human videos using a pre-trained transformer model to extract human poses. We will clarify in the revised manuscript.

Pre-training with action-free data is not a novel contribution.

We did not intend to claim that training with action-free data is unique to our work. Indeed, there has been prior work that studied this problem (we will include a discussion in the revised version of the manuscript). In this work, we show that a benefit of our approach is that it enables us to leverage action-free data for humanoid locomotion.

it appears that the robot can only walk randomly without controlling speed or turning. However, in Figure 5, it seems to control the robot’s walking. How is this achieved?

Our controller supports omni-directional walking at varying speeds of [-1,1] m/s. We will include additional videos in the revised supplementary materials. The walking speed and direction are specified as a 3-dim vector containing the desired linear velocity on x-axis and y-axis, and the desired angular velocity around the z axis. We specify the command as a part of the observation vector. For real-world deployment, we vary the command in real time via a joystick.

What exactly is the observation? It doesn't seem to be the observed images. How is the observation transformed into R^m?

The observation is a 102-dim vector which is the concatenation of the joint positions of each joint (26-dim), the velocities of each joint (26-dim), the linear and angular velocity of the body (6-dim), the gravity (3-dim), the clock input (2-dim), the command (3-dim) and the previous action containing the commanded position of each joint (20-dim) and the updated P and D gains for the PD controller (16-dim). The robot is blind and does not take visual information as input. Observations are projected using a single linear layer into the embedding dimension.

We will include the additional details in the revised version of the manuscript.

the authors did not conduct ablation experiments to verify the usefulness of each type of data.

Following reviewer suggestion, we perform the ablations below:

We see that each of the data sources leads to gains in performance individually and in aggregate.

The paper does not specify the control frequency or how the control frequency changes with model size, nor does it explain how to deploy the robot in the field. Where is the GPU located?

For all our experiments and all the model sizes we try in this work (1M, 2M, and 8M), the model is run at 50Hz during inference. The robot does not have a GPU on board and the model is run on the CPU of the on-board Intel NUC computer. In all of our outdoor experiments, everything is running on board. We use the 2M model by default for all experiments.

Related Work (GR1).

Thank you for the reference. While we focus on a different task (humanoid locomotion vs. manipulation) and use a different training strategy (staged training vs. joint training), there are some shared observations such as predicting observation can help action prediction. We will include the discussion in the revised version of the paper.

Writing and formatting.

Thanks for the suggestions. We will incorporate.

How to control the robot's walking? What exactly is Observation, and how is it transformed into R^m?

Please see our comments above.

From my understanding, the Neural Net Controller is the RL policy, the Model-based Controller is a scripted policy (non-learning method), and MoCap is a robot dataset, but not for the used Agility Robotics. Is my understanding correct?

Neural network controller is an RL policy from [33]. Please note that the Model-based Controller is a state-of-the art classical controller from Agility Robotics (for a dynamic task like humanoid locomotion it would be hard to script a policy). MoCap is not a robot dataset but a dataset of humans from [28, 24] (we retarget the human pose trajectories to humanoid locomotion trajectories via inverse kinematics).

One key limitation of the article is the lack of visual input, which is not discussed in the paper.

We focus on blind locomotion in this work which is a challenging problem. We intend to extend our approach to vision in future work. We will include a discussion in the revised manuscript.

Another limitation of the article is that it does not show how the real humanoid turns;

Our controller supports omni-directional walking and can turn in any direction. Please see above for additional discussion. We will include videos in the updated version.

Is it compliant to use YouTube videos, especially those featuring humans as the main subject?

The videos we use are a subset of public videos from research datasets (Kinetics, PoseTrack) and we pre-process the videos to extract poses which anonymizes the individuals in the videos.

Thank you for your response and for raising the score. Please find our responses below:

The project page doesn't include videos of the robot turning; can it actually turn in the real world or walk for an extended period of time? However, the project page seems to be inaccessible now. It would be interesting to see its obstacle avoidance capabilities under control.

Our policy can turn in all directions and walk for extended periods of time, up to 2 hours on a single battery charge. We have extensively tested these capabilities during a week-long deployment in San Francisco. Thanks to the responsive and accurate command following, our robot can avoid obstacles and navigate busy city streets with pedestrians.

As an additional data point, our omnidirectional walking capabilities are comparable to the state-of-the-art RL approach in Figure 3 of [33], with improved command following as demonstrated in Figures 5 and Figure 6 (left) of our manuscript.

We apologize for the inaccessible project page. It seems that our GitHub-hosted anonymous website was suspended by GitHub, which we are working to resolve with GitHub. Per the rebuttal instructions, it seems that we are not allowed to provide a new link to external pages in the rebuttal. We welcome any suggestions on how we might share the videos demonstrating these capabilities within the constraints of the rebuttal process and commit to include them in the release.

Additionally, if the robot needs to go uphill or downhill slightly, would it fail? And it will be fine to analyze its robustness.

Our policy can handle gentle slopes, and we have tested it in the real world on inclines of up to 8.7% (5 degrees). For example, Figure 1 (row 3, column 1) shows the robot walking up a 7% slope in San Francisco.

Regarding robustness, our approach trained purely on offline trajectories without any domain randomization has shown surprising effectiveness in walking in the real world. However, as we acknowledge in the limitations section, our policies may be less robust to sudden, large disturbances compared to RL policies.

In follow-up experiments, we found that pre-training policies with our proposed framework and fine-tuning them with a small amount of RL leads to excellent results, enabling rapid acquisition of new capabilities like walking on steeper slopes and robustness to large disturbances.

If this method were extended to enable humanoid robots to go up and down stairs, what challenges would arise?

To extend our method to enable the robot to go up and down stairs, we would need to incorporate vision to guide the foot placement. Our method can be readily extended to achieve this by incorporating images (potentially pre-processed using a vision encoder) as part of observations in addition to proprioception.

One challenge would be in collecting training trajectories that contain visual inputs. This would be relatively straightforward for trajectories collected using the same robot body (e.g., via prior RL or MPC controllers). To incorporate human trajectories from mocap or videos, we would need to include ego vision inputs where available (e.g., first-person videos) and use our strategy for training with missing data for trajectories without ego vision (e.g., third-person videos).

Some of the suggestions may go beyond the scope of this paper

Thank you for raising these questions. We will include the discussion of capabilities, limitations, and future work in the revised version of the manuscript with an extended discussion section.

Thank you for your response. We are glad to hear that we have addressed your remaining concerns regarding robot capabilities and appreciate your acknowledgement of our results.

We are more than willing to include a more detailed autoregressive models for robotics section. We will incorporate the suggestion in the revised manuscript.

We believe that we have addressed all of your concerns from the initial review and subsequent discussions. Given these and in light of the strengths you and all other reviewers have highlighted, we respectfully ask you to consider increasing the overall score to better reflect our contributions.

Thank you for the comments and suggestions! Please find our responses below:

In Sec. 3.6 Model inference, the first step might not follow the statement.

Thanks for pointing this out. Indeed, at the first step we have the current observation but not the action from the previous step. We use zero padding to fill in the missing actions for the initial step in both training and inference. We will clarify this in the next version of the paper.

It lacks description on the inference frequency

During inference, our neural network model predicts desired joint positions as well as PD gains at 50 Hz. These are used as targets for a low-level PD controller that runs at 2000 Hz.

It might be beneficial to ablate on how different sources of the data affect the performance

Thank you for the suggestion. We perform detailed ablations below:

We find that each of the data sources leads to gains in performance individually (first table) and in aggregate (second table).

In Table 1(c), why no Pred. Err. for staged training?

For stage training, the model is pre-trained to predict observations in the first stage, and then fine-tuned with action prediction in the second stage. Since the prediction error counts in both observation prediction and action prediction, and the model is not supervised with observation prediction in the second stage, it is probably not fair to compare with the model that is trained jointly on observation and action prediction. For reference, the prediction error for stage pre-training is 42.22 which is considerably higher than the prediction error of joint training (0.88).

In Sec. 4.3, can the authors detail the inverse kinematics problem they solve?

We formulate an inverse kinematics optimization problem as follows:

$ \min_{\substack{\mathbf{q}[t], \mathbf{\dot{q}}[t]}} ~ & \sum_{t=1}^{N} \varphi^{\text{traj}}[t] + \varphi^{\text{reg}}[t] \\ \text{s.t.} ~ & \mathbf{q}[t+1] = \mathbf{q}[t] + \frac{\mathbf{\dot{q}}[t+1] + \mathbf{\dot{q}}[t]}{2} dt, \\ & \mathbf{q} \in \mathcal{Q}, \mathbf{\dot{q}} \in \mathcal{V} $

where $\mathbf{q}$ is the robot state in the generalized coordinates, and $N$ and $dt$ are the optimization horizon and sampling time. The optimization variables include $\mathbf{q}$, $\mathbf{\dot{q}}$. For constraints, we include the Euler integration of posture $\mathbf{q}$, constrain the range of $\mathbf{q}$ and $\mathbf{\dot{q}}$ to their admissible sets $\mathcal{Q}$ and $\mathcal{V}$. In the cost function, $\varphi^{\text{traj}}$ tracks keypoint locations from human trajectories, and $\varphi^{\text{reg}}$ represents the regularization costs that include joint velocity minimization and smoothness.

In Sec. 4.4, can the authors detail on the filter strategy?

We empirically set a threshold on the reconstruction loss of the inverse kinematics retargeting and filter out the trajectories below the threshold.

How many Internet data are used? This seems not stated in the paper.

We use 1k human videos, which is the same number of trajectories we use for MoCap data.

It would be helpful to elaborate on current results of using Internet videos.

Please see the additional ablations reported above. We believe that this is a promising signal for using video data.

Thank authors for the detailed response. My questions are mostly addressed. My concern left is that will the largest model work at 50Hz, i.e. whether the increased model size with generative pretraining, hinders the inference. It would be great to discuss the minimum required frequency for functioning, and its relation to model size.

I have raised the score according to the detailed open-source commitment of authors.

Thank you for your response and for increasing the score. We think that the challenge of balancing model size and inference speed in real-time settings like ours is very interesting. Please find some comments below.

The minimal frequency for functioning is determined by the highly dynamic nature of the problem (unstable bipedal robot with a big upper body with a lot of mass and inertia). We might be able to reduce it a bit (e.g., 40Hz) but probably not much. Our current models are relatively small (up to 8M parameters) and can run within 50Hz on the on-board CPU computer.

We think that it is likely that the improvements in inference software (e.g., low precision inference), on-board compute (e.g., access to a GPU), training recipes (e.g., distillation), and modeling (e.g., sparse architectures) will offset these challenges and enable us to keep using larger models within the inference speed constraints.

We find the modeling angle to be a particularly interesting direction for future work. For example, one could imagine architectures with more parameters than flops, where different parameters are activated at different frequencies and different parts of the model are executed asynchronously.

We will include the additional discussion and an analysis of the model size and inference speed in the revised manuscript. Thank you for the suggestions.