We find it encouraging that the reviewer finds that our paper is well-written and well-organized and that our experimental results are impressive.

“[Would] this method would perform effectively if a large amount of human data and only a minimal amount of robot data were collected on a completely new task.”

We assume that sufficient human and robot data are required for domain alignment. The scope of our work to show that in single task settings we can resolve the domain gap between human and robot for Superior transfer for in-domain tasks over other human data augmented baselines Generalization to new appearances (objects and scenes) Generalization to new behaviours and motions that are only seen in human data We do agree that multi-task learning may allow one to collect less robot data on a completely new task, however the study of this is beyond the scope of this work. Our vision is that existing efforts to collect broad and diverse robot data can be augmented by an even larger distribution of human data of similar tasks across different scenes since human data with wearable devices can be collected with ease using EgoBridge.

“How are the trajectories divided into different steps? Human experience data might be continuous, while teleoperated data and human experience data may have different step structures to complete a task. How are these differences handled?”

We thank the reviewer for this question. To clarify, we do not perform alignment at an episodic level. Instead, as described in L164-167 we align individual samples of (observation, action chunk) pairs from the human and robot dataset. An action chunk is a finite sequence of atomic actions. In our case, as mentioned in L234-235 in the paper, it is a sequence of future cartesian end-effector poses. Therefore, given a sampled mini-batch of human and robot (observation, action chunk) pairs, we identify the most similar samples with DTW and use that as soft supervision for OT to align latent embeddings computed from the samples in a differentiable manner.

The reviewer asks “In soft supervision, the most similar trajectory should have a low transport cost. However, each trajectory consists of many steps. How are the steps aligned between the trajectories? Do the trajectories have the same length? Then, we directly align step 1 in the human trajectory to step 1 in the robot trajectory?”

The human and robot trajectories have the same length of 100. DTW captures spatial similarity between two trajectories while allowing for differences in execution speed. Each timestamp in the first trajectory is mapped to a corresponding (but potentially different) timestamp in the second and the distance is aggregated across timestamps to produce a cumulative loss. This is described in L185-192 of the paper. Qualitatively, this heuristic cumulative loss works well in finding similar trajectories as seen in the Appendix Section I.

“If OT is considered a type of trajectory selection method, then we align the representations between the selected teleoperated and human data. Can we directly align trajectory $i$ with $i^{\ast}(j)$?”

OT is not a trajectory selection method in our method. Instead, it is a differentiable training objective which encourages the feature encoder to align the latent space of human and robot data (L171-176). This results in a coupling feature distribution which permits transfer of knowledge. The reviewer astutely observes that it may be possible to directly align the positive pairs through deterministic mapping. This is precisely an alternate deterministic formulation of Optimal Transport Plan, the Monge map. We choose to use the probabilistic “soft” version of OT (Kantarovich) [27] as in a mini-batch setting, it is robust to pairing / label noise, and works better with behavior cloning since it retains the marginal distributions of human and robot data distributions. This design decision has also been inherited from related work in joint domain adaptation that leverage OT [1, 2, 3].

“Teleoperated data is certainly more challenging to collect. In the experiments, is the teleoperated data smaller than the human data?”

Yes, collecting teleoperated data is much more challenging to collect and in the experiments the human data is much larger than the teleoperated data. This is summarized in the Appendix Table 4, where, for the Drawer task, human data is 2.5x the amount of robot data (360 demo v/s 144 demo), the Scoop Coffee task, human data is 14x the amount of robot data (720 demo v/s 50 demo) and Laundry task, human data is 2.33x the amount of robot data (700 demo v/s 300 demo).

References

[1] Courty, N., Flamary, R., Habrard, A., & Rakotomamonjy, A. (2017). Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30.

[2] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation, 2016

[3] Damodaran, B. B., Kellenberger, B., Flamary, R., Tuia, D., & Courty, N. (2018). Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV) (pp. 447-463).

We thank the reviewer for their review. We are glad that they acknowledge our strong empirical results and think that our technical design is well motivated.

“[EgoBridge] necessitates collecting human demonstrations using specialized equipment with careful calibration to ensure action space alignment, semantically similar task execution, and comparable environmental conditions between human and robot domains.”

In regards to “using specialized equipment with careful calibration”, XR / smart wearables are growing ubiquitous. Our work simply requires some form of relative device localization at minimum. Many research devices such as Project Aria, and products such as Quest, Vision Pro with this capability are readily available. There are also many open source algorithms such as MANO and HAMER that provide 3D hand pose estimation. Large scale datasets with similar devices such as Nymeria, HOT3D, EgoExo4D, HOI4D, EgoDex etc. also leverage similar devices to collect human experience data.

Using camera frame to align action spaces is a very common design choice made in prior work [1, 4, 5]. In addition hand-eye calibration to obtain the transformation between base and camera frame is a very mature and common optimization based computer vision technique which can be applied trivially on any robot embodiment and dataset.

“when switching to a different robot embodiment, a completely new human dataset may be needed with embodiment-specific calibration and task adaptation.”

We respectfully disagree with the comment. To clarify, the human data we collect is general and does not require any robot-specific setup or calibration. As stated in L214-224, the data consists of first-person RGB images and proprioceptive hand tracking poses obtained directly from the Project Aria glasses. None of these assume a specific robot embodiment for datasets construction in the co-training setting. Task adaptation is also not required since we do not constrain how the human data is collected. Hence, our human data can be directly applied to co-train with data from any robot embodiment.

“comparable environmental conditions between human and robot domains”

As stated in L289-291 and shown in Figure 4 of the paper, we collect human data on an entirely different table, lighting conditions and background for the observation generalization result where EgoBridge generalizes to a completely new environment seen only in human data.

While we do agree the paper would benefit from “evaluation across different robot embodiments”, we do not have access to collect data and perform evaluation on another robot within the time constraints of the rebuttal.

We thank the reviewer for pointing out presentation issues, which we will address in the next revision of the paper. Specifically we will replace the bottom right image of Figure 3 with the bottom center image. We will also replace the mislabelled bottom center case with purple background without a mirrored T to match the described evaluation cases.

“Why does the OT cost only apply scalar reduction to the minimum DTW pairs?”

We thank the reviewer for their insightful question. We tried graduated scaling based on $A_{ij}$ by directly adding the $A_{ij}$ distance to the OT cost following prior work [6]. However, we found that this was unstable in training. We hypothesize that this is because of the large label shift between human and robot domains which makes the ordering of $A_{ij}$ noisy. Therefore we proposed a conceptually simple formulation that we also found to be stable during training empirically.

“For the Drawer experiments, how do the following factors affect performance: (1) varying the proportion of OOD drawers beyond 1/4, (2) different spatial arrangements of OOD drawers (random vs. clustered), and (3) different overlap ratios between human and robot training data?”

From empirical observation and prior work that studies spatial generalization [2, 3], random drawer arrangements would allow the policies to interpolate over the trajectories, especially when using a variational model like Flow matching. Hence we deliberately designed a challenging setting where the test region is held out in robot data to show a clear impact of human data. As shown in both DemoGen [2], this setting is strictly harder than a random configuration. Due to time constraints of the rebuttal, we are not able to collect additional data for the specific requirement and perform training and evaluation.

“What is the Laundry task performance under out-of-distribution scenarios such as different tables, unseen shirt types, or other environmental changes?”

To address this question, we provide an additional observation generalization experiment for the Laundry task. Here, in addition to the robot demonstrations, we add 30 minutes of human data on a new table with white surface unseen in robot data (we used a wooden table with brown color in robot data). We evaluate the co-trained baseline and EgoBridge on the new table. We perform 20 rollouts across 5 randomly sampled positions on the workspace. We calculate success rate and sub-task points as described in L292-297 of the paper. The results (success rate and points) are summarized in the table below.

In addition, we perform an additional experiment, where we test a more challenging case of combinatorially generalizing to a new shirt color on the white surface where the new shirt is seen on the original table in the human data and excluded from robot data entirely. We use an identical evaluation protocol all the previous experiment and report the results in the table below.

“Given the focus on scalable human data, what is the relationship between human data proportion and robot policy performance? Also please compare with other methods across varying human-robot data ratios?”

Prior work (EgoMimic [1]) with a similar co-training setup show a favorable scaling law for human data as compared to teleoperated robot data. We anticipate our method to show a similar or superior scaling law as we build on top of EgoMimic and show favorable performance across all evaluated tasks compared to it. We would like to support this with empirical results, however we are unable to perform this evaluation within the time constraints of the rebuttal.

References

[1] Kareer, S., Patel, D., Punamiya, R., Mathur, P., Cheng, S., Wang, C., ... & Xu, D. (2024). Egomimic: Scaling imitation learning via egocentric video. arXiv preprint arXiv:2410.24221.

[2] Xue, Z., Deng, S., Chen, Z., Wang, Y., Yuan, Z., & Xu, H. (2025). Demogen: Synthetic demonstration generation for data-efficient visuomotor policy learning. arXiv preprint arXiv:2502.16932.

[3] Saxena, V., Bronars, M., Arachchige, N. R., Wang, K., Shin, W. C., Nasiriany, S., ... & Xu, D. (2025). What matters in learning from large-scale datasets for robot manipulation. arXiv preprint arXiv:2506.13536.

[4] Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023). Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13778-13790).

[5] Wen, C., Lin, X., So, J., Chen, K., Dou, Q., Gao, Y., & Abbeel, P. (2023). Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025.

[6] Damodaran, B. B., Kellenberger, B., Flamary, R., Tuia, D., & Courty, N. (2018). Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV) (pp. 447-463).

[1] Meta Research, “Basics — project aria docs,” https://facebookresearch.github.io/projectaria tools/docs/data formats/mps/mps summary, 2024, accessed: September 15, 2024

We thank the reviewer for their positive review. We are encouraged that they find our results impressive and find the paper to be well written.

“The reliance and generalizability on DTW for action similarity may become impractical in multi-task or more diverse data regimes”

We thank the reviewer for the insightful comment. We acknowledge that this is a valid concern especially in multi-task settings where DTW-based action pairings may lead to incorrect or degenerate pairings, as discussed in L365-368 of the main text. However, the generality of our joint domain adaptation formulation allows for the inclusion of additional heuristics and labels. For instance, language embedding distances (as explored in adjacent works[1]) and dense language annotations of actions are becoming the norm in large scale imitation learning and would allow us to extend EgoBridge to more complex and multi-task settings in future works.

“What is preventing the tasks to get 80-90% success rates?”

We deliberately chose challenging long-horizon tasks with considerable reset variation as these are challenging settings that require a large amount of teleoperated demonstrations and hence can benefit the most from human data. The laundry task involved manipulation of a deformable object, the coffee task involves fine-grained grasping of a scoop with considerable reset variation and the drawer task requires very precise placement of the object. We could further improve the performance with additional robot data, but herein lies the unique benefit of EgoBridge of leveraging human data in resource-constrained settings, where we can get 70+% success in challenging tasks such as Laundry with only 300 demonstrations across a large workspace.

“Currently the human data is not very diverse. If human data collection can be scaled up and includes more diverse scenarios, would the policy be able to generalize to scenarios neither seen in human or robot data?”

Our work focuses in establishing joint domain adaptation as a method for imitation learning to align data from robot and human domains to enable transfer of knowledge from the source (human) to target (robot) domain. As such, we chose to focus our experiments on validating and studying the transfer of knowledge and whether the policy can generalize to scenarios only shown in human data. While achieving zero-shot generalization to new scenarios is an important problem for the field, we consider it to be an orthogonal direction that would require a dedicated, full-fledged research study.

“Can the policy generalize to objects of different shapes and sizes (not only appearance), that require different contact modes?”

Grasp primitives are not captured in the action space (cartesian). Hence given the large embodiment gap (parallel jaw v/s dextrous) in grasp, we hypothesize that grasping new object shapes will be challenging.

Can the policy generalize to new strategies or modes only seen in human data?”

We thank the reviewer for this question. We acknowledge that generalization to new modes is a challenging task. We show this in EgoBridge, through the drawer task where the robot needs to place the object in target drawers where there is no spatial robot data, hence achieving motion generalization through human data. Future work will study other modes and behaviours we can transfer from human data.

“How expensive is the OT computation per training step? Does it impact training scalability in larger datasets?”

Thanks for the clarification question. We use the geomloss [2] library which is a kernel optimized sinkhorn process that runs in parallel on GPU. The run-time of this algorithm is $O(N^2)$ where N is the batch size. In practice OT computation is a fraction of the time compared to the model forward pass. To empirically validate this claim, we use a single L40s GPU, with batch size 32 and torch.cuda.synchronize() to profile over 10 batches.

Average forward pass time (human + robot) : 0.344373s
Average OT Loss time : 0.005504s

Which makes the $T^{\ast}$ computation roughly 1.6% of the forward pass time and hence it doesn’t impact training scalability with larger batch sizes and datasets.

“Have you compared to contrastive or VLM-based alignment methods as an alternative to OT?”

Yes, we have experimented with contrastive-based alignment methods as an alternative. We empirically observed that when naively applied, contrastive learning results in unstable training. Specifically, we utilized the same DTW action supervision to form positive pairs and used the InfoNCE [3] loss on policy latents. We hypothesize that this instability is a result of contrastive learning not retaining the marginal distributions of the human and robot, which is crucial for behavior cloning. Additional research is required to correctly adapt contrastive learning for joint domain adaptation.

“How critical are the wrist camera inputs in robot data compared to the egocentric view? Would EgoBridge still be effective without them?”

We thank the reviewer for the insightful comments. We had a similar hypothesis that removing wrist camera may improve alignment, but we empirically found that the wrist camera is critical to manipulation steps that require precision, such as grasping and alignment, as echoed by many other research [4, 5] . Hence we designed the architecture to handle heterogeneous sensor modalities from the human and robot data.

We thank the reviewer for identifying presentation issues, which we will address in the next revision of the paper. Specifically, we will align the arrows in Figure 2, replace the purple legend with Human + Robot, remove the robot legend and make the colors consistent in Figure 4.

References

[1] Yu, A., Foote, A., Mooney, R., & Martín-Martín, R. (2024). Natural language can help bridge the sim2real gap. arXiv preprint arXiv:2405.10020.

[2] Feydy, J., Séjourné, T., Vialard, F. X., Amari, S. I., Trouvé, A., & Peyré, G. (2019, April). Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd international conference on artificial intelligence and statistics (pp. 2681-2690). PMLR.

[3] Oord, A. V. D., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

[4] Liu, Y., Romeres, D., Jha, D. K., & Nikovski, D. (2020). Understanding multi-modal perception using behavioral cloning for peg-in-a-hole insertion tasks. arXiv preprint arXiv:2007.11646.

[5] Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., ... & Martín-Martín, R. (2021). What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298.

We thank the reviewer for their positive evaluation of our proposed method. We are encouraged that they found our method clever and results promising!

"For example, consider two tasks of picking a cup and opening a drawer. If the cup and the drawer are at similar position, then most part of the trajectories is very similar (the part of reaching to the object) and maybe only the last part of the trajectories (how to manipulate the object) is different."

We thank the reviewer for the insightful comment. We acknowledge that this is a valid concern especially in multi-task settings where DTW-based action pairings may lead to incorrect or degenerate pairings, as discussed in L365-368 of the main text. However, the generality of our joint domain adaptation formulation allows for the inclusion of additional heuristics and labels. For instance, language embedding distances (as explored in adjacent works[1]) and dense language annotations of actions are becoming the norm in large scale imitation learning and would allow us to extend EgoBridge to more complex and multi-task settings in future works.

"is it costly to solve the optimization problem and find $T^{\ast}$?

Thanks for the clarification question. We use the geomloss [5] library which is a kernel optimized sinkhorn process that runs in parallel on GPU. The run-time of this algorithm is $O(N^2)$ where N is the batch size. In practice finding $T^{\ast}$ is a fraction of the time compared to the model forward pass. To empirically validate this claim, we use a single L40s GPU, with batch size 32 and torch.cuda.synchronize() to profile over 10 batches.

Average forward pass time (human + robot) : 0.344373s
Average OT Loss time : 0.005504s

Which makes the $T^{\ast}$ computation ~1.6% of the forward pass time.

“when we are updating the model during training, the distribution of the latent representation is changing and thus the $T^{\ast}$ might also change. How to we track the changing optimality of $T^{\ast}$?”

We thank the reviewer for the insightful comment. $T^{\ast}$ computed on the policy latent is indeed evolving with the training process. $T^{\ast}$ is computed for each batch at each training step. Since we don’t use $T^{\ast}$ as a push forward but instead as a differentiable objective guiding the direction of moving probability mass from the two distributions, it acts as the supervision to the encoder to move the latent representations such that it reaches a cost-minimizing $T^{\ast}$ for all sampled batches across the dataset.

Adjacent to this, the reviewer asks “What if we directly use it as a hard supervision instead of soft supervision? For example, for each human demo, we find the robot demo that has the highest similarity to it, and then directly minimizes the distance between their latent representation?”

This is precisely an alternate deterministic formulation of Optimal Transport Plan, the Monge map. We choose to use the probabilistic “soft” version of OT (Kantarovich) [27] as in a mini-batch setting, it is robust to pairing / label noise, and works better with behavior cloning since it retains the marginal distributions of human and robot data distributions. This design decision has also been inherited from related work in joint domain adaptation that leverage OT [2, 3, 4].

References

[1] Yu, A., Foote, A., Mooney, R., & Martín-Martín, R. (2024). Natural language can help bridge the sim2real gap. arXiv preprint arXiv:2405.10020.

[2] Courty, N., Flamary, R., Habrard, A., & Rakotomamonjy, A. (2017). Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30.

[3] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation, 2016

[4] Damodaran, B. B., Kellenberger, B., Flamary, R., Tuia, D., & Courty, N. (2018). Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV) (pp. 447-463).

[5] Feydy, J., Séjourné, T., Vialard, F. X., Amari, S. I., Trouvé, A., & Peyré, G. (2019, April). Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd international conference on artificial intelligence and statistics (pp. 2681-2690). PMLR.