Pre-training Auto-regressive Robotic Models with 4D Representations

We thank the reviewer for the insightful comments. We address the concerns raised below:

1. Causal Analysis & Optimality of 4D Representations: As stated in the paper, one of the main benefits of using 4D representations is that the 3D point tracks in a robotic setting are described by linear transformations constructed using the robot states. We present a simple proof of this below:

Consider a $n$-DoF open-chain manipulator, initially in its reference configuration $\theta_1=0, \cdots, \theta_n=0$. Suppose the manipulator is commanded to a new position $\theta_1=\theta_1^d, \cdots, \theta_n=\theta_n^d$. Next, we take any arbitrary point $p$ on the body of the manipulator lying between joints $i, i+1$. Let $p$ be described by the transformation $g_{i,p} (0)$ in joint $i$'s frame. Then, $p$'s new position in the base frame can be described by:

$$g_{1,p} (\theta) = e^{{\hat{\xi_1}}\theta_1^d}\cdots e^{{\hat{\xi_i}}\theta_i^d} g_{i,p} (0)$$

where ${\hat{\xi_j}}$ are the twists associated with each joint. Since this product of 4x4 SE(3) matrices represents a linear transformation, points on the robot body as well as attached rigid bodies evolve through linear transformations in terms of the robot states.

Overall, this proof shows that learning 3D point track prediction is related to and can benefit robotic control, as validated by our ablation experiments. Additionally, for overactuated manipulators (7 DoF), the robot learning problem simplifies to tracking arbitrary trajectories, as configuration space limitations are less restrictive. This allows the model to benefit more effectively from the 3D point tracking pre-training task.

2. Comparison to 2D and 3D methods: We note that in our paper (see Tables 1/2), we compare ARM4R with both 2D and 3D approaches. ARM4R outperforms 2D motion-based methods (e.g., LLARVA, ATM) and 3D-based methods (e.g., ManiGaussian). We also include an additional comparison below with a recent 3D method, 3D-VLA, further demonstrating the strength of 4D representations for robotic learning.

3. Geometric Alignment: We clarify that our claim regarding geometric alignment refers not to alignment between human and robotic states, but between 3D point tracks in a robotic setting and the robot states. As shown in the proof above, these point tracks are described by linear transformations of the robotic states, which suggests that 4D representations are a natural choice for robotic control. We will clarify this in the final version.

4. 2D-to-3D Lifting: An alternative to monocular depth estimation is to use multiview cues and triangulation, which is more appropriate for dynamic camera setups. However, since our deployment in both real-world and simulation environments involves static cameras, monocular depth estimation is more suitable. This is the common and simple practice used in many 3D tracking papers, such as SpatialTracker and SoTA method DELTA.

5. Scalability: As the reviewer suggested, we add a scaling experiment, where we change the number of demonstrations used in our Epic-Kitchens pre-training. Specifically, we decrease the number of human videos in the pretraining to ⅓ and ⅔ of the original dataset used. We train the model for all three stages, and evaluate the performance on the three cube tasks. The results are summarized below:

The results show that as the size of the pre-training set increases, the performance also tends to increase.

6. Real Robot Evaluation: We note that the 3 tasks of pick, stack and destack in Table 3 are commonly used for evaluation in several baselines, including LLARVA, Octo, RPT, and MVP. The main reason we follow the same procedure in this experiment is that comparing ARM4R to 6 different pre-training approaches on all 13 tasks would require a significant amount of demonstrations, as every baseline would need to be fine tuned for a fair comparison. Specifically, fair fine-tuning for each baseline would demand approximately four times the data currently used. We plan to expand the evaluation to include additional tasks in the final version.

7. Monocular Depth Estimation: See point 2 under Reviewer Zx5c.

8. Robustness Analysis: See point 3 under Reviewer vJYs.

9. Failure Case Analysis: See point 1 under Reviewer vJYs.

10. Generalization to Dynamic environments: See point 2 under Reviewer vJYs.

11. Dataset Bias: See point 4 under Reviewer vJYs.

We thank the reviewer for the insightful comments. We address the concerns raised below:

1. Additional Limitations: Here we provide a further analysis investigating more limitations of our approach.

(i) Unnatural rotation: We examined a new task, “put knife” in RLBench. Interestingly, this simulated task features an unnatural whole-arm rotation to grasp the knife handle, based on the expert demonstrations created by the simulator’s motion planning. As the movement between two key points is very long, we hypothesize that our model has difficulty learning this rotation.

(ii) Precision: We observed that ARM4R struggles with the 'screw bulb' task, a procedure requiring precise insertion into the bulb holder. In contrast, our model effectively performs other standard precision tasks, such as 'open drawer,' which necessitate only standard precision.

2. High Variability (Dynamic Environments) task performance: ARM4R can handle dynamic changes as it predicts each step in a trajectory, thus interacting with the environment continuously. To further evaluate dynamic variability, we test three cube tasks (stack, destack, and pick), manually shifting the cube during execution when the arm reaches ⅓ and ⅔ of its starting height. See the results:

Results show consistent performance despite target object movement, as long as sufficient z-axis space is available for adjustment. This highlights the model's robustness to environmental and object-level changes.

3. Robustness analysis: To assess the robustness of ARM4R regarding factors like camera noise, occlusions, or sensor drift, we perform additional evaluations on the three cube tasks under the following conditions: (1) Dim lighting: ambient light reduced to 50%, (2) Background distractors: dynamic background changes, such as people walking by or adjusting curtains, and (3) Tabletop distractors: two random objects placed near the target. The results are presented below.

Overall, ARM4R demonstrates robustness to lighting changes and background distractors, likely due to the attention pooling (See Fig 2 in the main paper), which guides the model to focus on the target region rather than background features. This is further supported by the observed performance drop when tabletop distractors are introduced.

4. Dataset Bias: We note that EpicKitchens is one of the main vision datasets containing various kinds of interactions between human hands and objects in kitchens. Thus, we choose to leverage it as a pre-training dataset for tabletop manipulation tasks. In future work, we plan to explore the benefits and effects of adding more pre-training datasets.

5. Domain Gap: As the reviewer correctly pointed out, there exists a domain gap between human video data and robotic control environments. To address this issue, we introduced in the paper Stage 2 of our training pipeline: Fine-tuning 3D Point Tracking for Robotic Settings. This stage uses the same task as in Stage 1 but enables the model to adapt to differences in camera dynamics and embodiment from the target robotic environment. The impact of this domain gap is evident in our results (Section 4.4, Figure 3), where omitting Stage 2 leads to decreased success rates. These findings underscore the importance of Stage 2 in bridging the gap and improving downstream performance.

6. Number of demonstrations used: In our real Kinova setting, we used 200 demonstrations (split into 19:1 training/validation sets) per task to finetune our model and all the other baselines except RPT and LLARVA in Tables 2 and 3. The RPT model uses 1920 demonstrations for its sensorimotor pre-training, and LLARVA uses 800 demonstrations per task. We note that for our newly added results on LLARVA, we used 200 demonstrations per task, following the same setting as the other baselines in Table 2. This may explain LLARVA’s poor performance on most of the tasks tested, as it typically requires more data. Overall, we found that training our model on 200 demonstrations is enough to achieve good performance in both real and simulation settings.

We thank the reviewer for the insightful comments. We address the concerns raised below:

1. Comparison to RVT, RVT-2: As the reviewer noted, RVT and RVT-2 are recent transformer-based methods that predict keyframes in robot trajectories. Importantly, both methods leverage RGB-D images to reconstruct a scene's point cloud and then predict keypoints. This point cloud reconstruction relies on a similar principle to our method, with ARM4R using pre-training on the 3D point tracking task to learn scene structure. We will add a proper discussion and comparison to this line of work in the final version of the paper.

However, we would like to highlight that our approach contains some significant differences compared to RVT/RVT-2. First, our approach leverages 3D point tracking from human video using depth estimation, relying on pseudo labels. This enables the utilization of readily available web-based video data. In contrast, RVT and RVT2 depend on ground-truth RGB-D data and reconstructed point clouds captured by cameras. This dependence on ground-truth depth data inherently restricts the amount of usable video data. In addition, our approach uses 3D point tracking that can be easily extracted from human video data compared to keyframe-based approaches which leverage heuristics such as velocity changes to find keyframes.

2. Depth Estimation Model: To obtain the 3D point tracks, we use SpatialTracker, which utilizes ZoeDepth [1] as the depth estimator. We note that since ZoeDepth predicts metric depth, it also predicts the absolute depth map. Additionally, as demonstrated in TapVid-3D [3]—a follow-up work of SpatialTracker—and DELTA [4], utilizing UniDepth [2], a more accurate monocular depth model, can enhance tracking performance. Lastly, as mono-depth methods further improve (such as UniDepth v2 [5]), 3D point tracking on monocular videos will also further improve.

[1] Bhat, Shariq Farooq, et al. "Zoedepth: Zero-shot transfer by combining relative and metric depth." arXiv preprint arXiv:2302.12288 (2023).

[2] Piccinelli, Luigi, et al. "UniDepth: Universal monocular metric depth estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Koppula, Skanda, et al. "Tapvid-3d: A benchmark for tracking any point in 3d." arXiv preprint arXiv:2407.05921 (2024).

[4] Ngo, Tuan Duc, et al. "DELTA: Dense Efficient Long-range 3D Tracking for any video." arXiv preprint arXiv:2410.24211 (2024).

[5] Piccinelli, Luigi, et al. "Unidepthv2: Universal monocular metric depth estimation made simpler." arXiv preprint arXiv:2502.20110 (2025).

We thank the reviewer for the insightful comments. We address the concerns raised below:

1. Table 1 Results: As noted by the reviewer, the simulation results for ARM4R are slightly skewed by the high success rate for the “put money” task. We recalculate the success rate without this outlier for ARM4R and the closest competitor, PerAct: Our model achieves a success rate of 56.51%, while Peract achieves a marginally lower success rate of 56.36%. We note that PerAct also has access to ground truth voxel information from the simulator, while our model learns 3D understanding from its own training, only using RGB images at inference time.

To further evaluate our model against other 3D-based approaches, we also compare it with 3D-VLA—a 3D-based VLA model that generates goal point clouds. Since the official training and inference codebase is not publicly available, we compare ARM4R to 3D-VLA using three RLBench tasks for which results are reported in the 3D-VLA paper. The results are shown as follows:

We see that ARM4R performs much better than all other tested models on 2/3 tasks, and the average performance is better as well. The poor performance on the put knife task is explained by the nature of data collected. This simulated task features an unnatural whole-arm rotation to grasp the knife handle, based on the expert demonstrations created by the simulator’s motion planning. Due to the large movement between the first two keyframes, we hypothesize that our model struggles to learn the rotation because the context window becomes saturated with local motion details, leading to a loss of global trajectory information. This is why LLARVA struggles with this task as well, since it has to predict the whole trajectory without skipping any step’s action. In contrast, 3D-VLA predicts keyframes and thus does not suffer from this issue. Finally, since our model is pre-trained on human data and this rotation is very unnatural, it might be more difficult to learn.

2. Table 2 Results: We add two more baselines to our real Kinova setting: (1) LLARVA, a 2D-based VLA, and (2) Pi0-FAST, a recent SoTA VLA model. The results are below:

Both models exhibit low performance: LLARVA / Pi0 achieves an average success rate of 18.3% / 21.2%. This is likely due to the amount of fine-tuning data used: LLARVA typically requires 800 episodes per task (we have here 13 tasks), Pi0-FAST shows good results when tuned for the whole DROID dataset (~76K demos), while we only have a total of 2,600 training episodes for our real setting. For robotics, collecting a large amount of demonstrations is expensive and difficult, so learning efficiency is a key performance indicator.

3. Table 4: We added an extra experiment for Franka robot which includes: (1) Kinova 3D points tracking stage (2) Franka robot joint fine-tuning. The results are shown as follows:

The results show that only training on Kinova 3D points tracking can also boost results when a Franka robot is used in the downstream setting, although the increase in performance is not as big as the one compared to when human video pre-training is also used. This aligns with the hypothesis that both Stage 1 and Stage 2 are important to achieve the best possible performance.

4. Dataset preparation: We provide details on human video dataset preparation in Appendix B1. Regarding camera motion, static cameras are a subset of dynamic cameras; thus, our robotic settings are not out of distribution with respect to the pre-training.

5. Additional Experiments: Due to space constraints, please refer to points 2 and 3 in response to reviewer vJYs for experiments on handling distractors and dynamic environments.