Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training

Thank you for your time and thoughtful feedback. We appreciate the positive recognition of our integrated co-training framework for sim-to-real transfer, which is both methodologically sound and innovative. We are also grateful for the acknowledgment of its ability to handle imbalanced data distributions across domains—a critical challenge in practical real-world applications—and its consistent performance gains in challenging evaluation scenarios, including real-world experiments, a setting often underexplored in prior work. Below, we address the raised concerns in detail:

• Sensitivity to hyper-parameters: We conducted an ablation study to evaluate the sensitivity of our method to key hyperparameters, including the entropy regularization coefficient ($\epsilon$), the KL divergence penalty term ($\tau$), and the window size used in temporally aligned sampling. In each experiment, we varied a single hyperparameter while keeping the others fixed, trained the policy, and assessed its performance via rollout evaluations. We report results for the BoxInBin task under the Viewpoint-Image setting and the Lift task under the Texture-Image setting, as shown in the following tables (Table R1-R6). The results indicate that our method is robust to hyperparameter variations within reasonable ranges. Specifically, performance remains stable when $\epsilon$ and $\tau$ are set between 0.001 and 0.1, and when the window size is varied between 5 and 20. Our method consistently outperforms the co-training baseline in out-of-distribution (OOD) scenarios, where the baseline achieves a success rate of 0.14 on BoxInBin and 0.6 on Lift. These findings suggest that, although our approach introduces additional components, it does not require extensive tuning and offers clear advantages in terms of generalization.

Table R1: Success rate of our method on the BoxInBin task when varying $\epsilon$.

Table R2: Success rate of our method on the BoxInBin task when varying $\tau$.

Table R3: Success rate of our method on the BoxInBin task when varying the window size.

Table R4: Success rate of our method on the Lift task when varying $\epsilon$.

Table R5: Success rate of our method on the Lift task when varying $\tau$.

Table R6: Success rate of our method on the Lift task when varying the window size.

• Effectiveness with extremely limited target domain data: We argue that in real-world robotic settings, 10–25 demonstrations constitute a highly limited amount of data—particularly when compared to prior works such as [1], which often require more than 100 demonstrations. This limitation is further compounded in our evaluation, which involves diverse object poses, textures, shapes, and extended temporal horizons. To address the reviewer’s concern, we provide a comparative analysis of different methods under this low-data regime in the target domain, specifically for the BoxInBin task with the Viewpoint3-Point setting. The results show that our approach consistently outperforms the baselines, demonstrating its robustness even with extremely limited supervision.

Table R7: Success rates and confidence intervals for various methods on the BoxInBin task with the Viewpoint3-Point setting, under scenarios where data from the target domain is extremely limited.

• Generalize to tasks with significant dynamics gaps: As stated in Line 119-121, our work focuses on quasi-static tasks, which are representative of many everyday scenarios in home environments. In these settings, the observation gap is the primary bottleneck for sim-to-real transfer and presents a substantial challenge on its own. While handling dynamics mismatches is not the central focus of this work, our method is built on a co-training framework, which inherently provides some robustness to dynamics gap. Recent works[2, 3] have also demonstrated that co-training strategies can mitigate the effects of dynamics gaps and handle certain dynamic tasks. We acknowledge this as a limitation of the current study and view extending our framework to more dynamic tasks as a valuable direction for future research.
• Alternative domain alignment methods: Our problem setting specifically targets sim-to-real transfer under out-of-distribution (OOD) conditions in the real world, with limited real-world demonstrations. The domain alignment methods mentioned by the reviewer[4, 5] primarily rely on adversarial training to learn a shared latent space for the transitions, but they do not account for the significant data imbalance between the source (simulation) and target (real-world) domains—a core challenge in our setting. While these approaches are relevant in broader domain adaptation contexts, they would require significant effort to be adapted to the problem formulation in this paper. We will cite and discuss these aspects in the related work section of the revised manuscript.
• Missing citations: We will cite and discuss these works[4, 5, 6] in the revision.
• Dependence on demonstration alignment: We would like to clarify the notion of "alignment" as used in our work. Our method assumes alignment only at the level of task semantics—that is, demonstrations across different domains are expected to operate on the same object geometry and pursue the same task objective. We do not assume identical execution rates or strict temporal synchronization between trajectories. To account for temporal variability, our framework incorporates a temporally aligned sampling strategy during training. Comparable or even more restrictive assumptions regarding semantic alignment have been adopted in prior work. For example, Raychaudhuri et al.[4] referenced by the reviewer assumes that linearly normalized temporal positions correspond to the same task semantics—an assumption that may not hold in real-world robotics scenarios, where execution speeds can vary significantly across domains and task stages. We contend that the semantic alignment assumption required by our method is practical and readily achievable, particularly given recent advances in perception systems[7], digital twin technologies[8, 9], and motion synthesis tools[10, 11] that enable the generation of diverse and semantically consistent demonstrations from a small number of source demonstrations.

[1] Natural Language Can Help Bridge the Sim2Real Gap, RSS'24;

[2] Empirical Analysis of Sim-and-Real Cotraining Of Diffusion Policies For Planar Pushing from Pixels, IROS'25;

[3] Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, RSS'25;

[4] Cross-domain Imitation from Observations, ICML'21;

[5] Domain Adaptive Imitation Learning, ICML'20.

[6] Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model, ACML'25

[7] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects, CVPR'24;

[8] Automated Creation of Digital Cousins for Robust Policy Learning, CoRL'24;

[9] Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation, RSS'24;

[10] Mimicgen: A data generation system for scalable robot learning using human demonstrations, CoRL'23;

[11] DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning, RSS'25.

Thank you for your time and constructive feedback. We appreciate the positive recognition of our work in proposing a conceptually compelling framework for addressing challenging real-world robotics problems, particularly under the practical constraint of limited real-world demonstrations—a setting that is often overlooked in prior work. Below, we address the raised concerns in detail:

• Theoretical or methodological contributions: We thank the reviewer for their feedback. Our work addresses a critical emerging challenge in robot learning: how to achieve real-world generalization when training primarily on simulated data with only sparse real demonstrations. While Optimal Transport (OT) has been used in domain adaptation, existing methods fail to handle the unique requirements of policy co-training—particularly the need to preserve action-relevant features across domains while coping with severe data imbalance ($N_{src}$ >> $N_{tgt}$) and partial state overlap. Our key innovations include: (1) joint alignment of observations and actions (rather than marginal features alone), (2) an Unbalanced OT formulation to handle domain mismatch, and (3) temporally-aware sampling for sequential data—all specifically designed for the sim-to-real policy setting. The problem setting itself represents an important and understudied challenge. As large-scale simulation systems like MimicGen[1] enable massive synthetic datasets, the fundamental question becomes how to effectively leverage this data for real-world generalization, especially for states unseen in real demonstrations. Our framework provides the first systematic solution, achieving +30% success rates on out-of-distribution real-world tasks compared to standard co-training (Table 2 and Table 4). This demonstrates that our methodological adaptations are not just applications of OT, but necessary innovations to bridge simulation and reality for policy learning. We highlight the growing paradigm of scaling imitation learning by combining simulated and real-world data[2, 3]. Recent advances in robot data synthesis[1, 4, 5] enable controllable, large-scale simulation. Our main contribution is to formalize and address the overlooked challenge of out-of-distribution (OOD) generalization in the target domain. We propose a novel OT-based framework that addresses data imbalance and mini-batch limitations while retaining theoretical guarantees, and validate it through real-world experiments under practical constraints—often neglected in prior work.
• Complexity in implementation and tuning: We thank the reviewer for pointing out the importance of evaluating the trade-off between algorithmic complexity and practical benefit. To address this concern, we conducted an additional ablation study to analyze the sensitivity of our method to its key hyperparameters, including the entropy regularization coefficient ($\epsilon$), the KL divergence penalty term ($\tau$), and the window size used in temporally aligned sampling. In each experiment, we varied a single hyperparameter while holding others fixed, trained the policy, and assessed performance via rollout evaluations. We report results for the BoxInBin task under the Viewpoint-Image setting and the Lift task under the Texture-Image setting, as shown in the following tables (Table R1-R6). The results show that our method is robust to a range of reasonable hyperparameter choices and does not require fine-grained tuning to achieve strong performance. Regarding the implementation complexity, while UOT and temporally-aware sampling do introduce some algorithmic overhead, both components are modular and can be easily integrated into standard BC pipelines. To further support reproducibility, we will release the codebase upon acceptance.

Table R1: Success rate of our method on the BoxInBin task when varying $\epsilon$.

Table R2: Success rate of our method on the BoxInBin task when varying $\tau$.

Table R3: Success rate of our method on the BoxInBin task when varying the window size.

Table R4: Success rate of our method on the Lift task when varying $\epsilon$.

Table R5: Success rate of our method on the Lift task when varying $\tau$.

Table R6: Success rate of our method on the Lift task when varying the window size.

• Effectiveness with extremely limited target domain data: We argue that in real-world robotic settings, 10–25 demonstrations constitute a highly limited amount of data—particularly when compared to prior works such as [6], which often require more than 100 demonstrations. This limitation is further compounded in our evaluation, which involves diverse object poses, textures, shapes, and extended temporal horizons. To address the reviewer’s concern, we provide a comparative analysis of different methods under this low-data regime in the target domain, specifically for the BoxInBin task with the Viewpoint3-Point setting. The results show that our approach consistently outperforms the baselines, demonstrating its robustness even with extremely limited supervision.

Table R7: Success rates and confidence intervals for various methods on the BoxInBin task with the Viewpoint3-Point setting, under scenarios where data from the target domain is extremely limited.

• More recent and relevant sim-to-real baselines: Our problem setting specifically focuses on sim-to-real transfer under out-of-distribution (OOD) conditions in the real world, with limited access to real-world demonstrations and no prior knowledge of the target distribution. While recent methods in robotic sim-to-real transfer have shown progress[2, 3, 7, 8, 9], most do not directly tackle this challenging OOD generalization scenario. For example, some approaches require human-in-the-loop interventions for online data collection after deployment[8], others rely on computationally expensive reinforcement learning pipelines[7], or adopt domain randomization[9], which assumes knowledge of the target domain distribution and may not generalize well when the test-time conditions fall far outside the training distribution. In contrast, our setting addresses scenarios where the target domain is not fully known a priori, and the task objective may vary or fall outside the training distribution, making traditional sim-to-real pipelines less suitable. Among these works, co-training approaches[2, 3]—which train policies using a mixture of simulation and real-world data—are most aligned with our problem setting and have demonstrated strong performance. Therefore, we focus on these methods as baselines, as they are the most relevant to our formulation and constraints, ensuring meaningful and fair comparisons. We appreciate the reviewer for highlighting additional related works.
• Detailed information on real-world datasets: We provide details of the real-world dataset in Section B.3 of the supplementary material. All baselines and our method are trained on the same dataset, ensuring a fair and consistent comparison.

[1] Mimicgen: A data generation system for scalable robot learning using human demonstrations, CoRL'23;

[2] Empirical Analysis of Sim-and-Real Cotraining Of Diffusion Policies For Planar Pushing from Pixels, IROS'25;

[3] Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, RSS'25;

[4] Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning, ICRA'25;

[5] DemoGen: Synthetic Demonstration Generation for Data-Efficient Visuomotor Policy Learning, RSS'25;

[6] Natural Language Can Help Bridge the Sim2Real Gap, RSS'24;

[7] From Imitation to Refinement -- Residual RL for Precise Assembly, arXiv'24;

[8] TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction, CoRL'24;

[9] DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality, ICRA'23.

Thank you for your valuable comments. As the discussion deadline approaches, we would like to confirm if our latest response addresses all your concerns.

Thank you for your time and thoughtful feedback. We are grateful for the positive recognition of the novelty of our work, the high-quality accompanying video, and the thorough evaluations conducted across diverse modalities and domains, particularly under challenging out-of-distribution (OOD) conditions. Below, we address the raised concerns in detail:

• No strong real-world baseline/oracle: We interpret the reviewer’s concern regarding the “40% success rate for the real-world Stack task” as referring to the results in Table 2, which correspond to the out-of-distribution (OOD) generalization setting—specifically, cube placements in novel poses not seen during training. These OOD configurations are intentionally challenging and designed to evaluate a model’s ability to generalize beyond observed real-world data. To provide additional context, we also report in-distribution (ID) performance for all methods, where our approach achieves a 90% success rate (see Table 6 in the supplementary). This sharp contrast between ID and OOD performance confirms the validity of our hardware setup and data collection process, while underscoring the core challenge of sim-to-real generalization under OOD conditions—the primary focus of our work. Regarding the absence of a strong real-world oracle, we emphasize that our work is intentionally situated in the low-data regime. This reflects the practical difficulty of collecting large numbers of real-world demonstrations across diverse OOD scenarios. For example, training a diffusion policy solely on real-world data with sufficient OOD coverage would typically require hundreds of demonstrations[1]—an expensive and often impractical requirement. Our setting is deliberately designed to reflect these real-world constraints. Instead of relying on extensive real-world data, we explore how simulation data can be effectively leveraged—combined with limited real-world supervision—to bridge the sim-to-real gap under OOD target distributions.
• Reproducibility: To enhance the reproducibility of our work, we plan to include code examples illustrating the implementation of the core components in the revised version. We also intend to release the codebase and dataset upon acceptance.
• Sensitivity to hyper-parameters: We conducted an ablation study to evaluate the sensitivity of our method to key hyperparameters, including the entropy regularization coefficient ($\epsilon$), the KL divergence penalty term ($\tau$), and the window size used in temporally aligned sampling. In each experiment, we varied a single hyperparameter while keeping the others fixed, trained the policy, and assessed its performance via rollouts. We report results for the BoxInBin task under the Viewpoint-Image setting and the Lift task under the Texture-Image setting, as shown in the following tables (Table R1-R6). The results indicate that our method is robust to hyperparameter variations within reasonable ranges. Specifically, performance remains stable when $\epsilon$ and $\tau$ are set between 0.001 and 0.1, and when the window size is varied between 5 and 20. Our method consistently outperforms the co-training baseline in out-of-distribution (OOD) scenarios, where the baseline achieves a success rate of 0.14 on BoxInBin and 0.6 on Lift. These findings suggest that, although our approach introduces additional components, it does not require extensive tuning and offers clear advantages in terms of generalization.

Table R1: Success rate of our method on the BoxInBin task when varying $\epsilon$.

Table R2: Success rate of our method on the BoxInBin task when varying $\tau$.

Table R3: Success rate of our method on the BoxInBin task when varying the window size.

Table R4: Success rate of our method on the Lift task when varying $\epsilon$.

Table R5: Success rate of our method on the Lift task when varying $\tau$.

Table R6: Success rate of our method on the Lift task when varying the window size.

• The importance of sampling strategy: Although Optimal Transport (OT) can theoretically align samples across different task stages when full episodes are used, this approach is often impractical due to computation constraints. In mini-batch training with limited batch sizes, especially for multi-stage or long-horizon tasks, semantically aligned samples may rarely co-occur within the same batch. Our sampling strategy addresses this by increasing the likelihood of including aligned pairs, thereby improving alignment quality and accelerating convergence. As demonstrated in Figure 9, this approach leads to significant improvements in policy performance.

[1] DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, RSS'24.

Thank you for your time and constructive feedback. We sincerely appreciate the positive recognition of our work’s clear motivations and contributions, the sufficient implementation details provided, and the well-designed experiments validating these contributions. Below, we address each of the concerns in detail:

• Statistical guarantee from the sim-to-sim experiments: For each task and method, we perform 50 policy rollouts with randomly sampled initial states, following standard protocol in prior works [1, 2, 3], where the number of trials is generally considered sufficient to ensure statistical significance. We additionally report 95% confidence intervals using standard error in Table R1 and Table R2, with the criterion used in prior works[4], for all simulation experiments. In our tables, bolded numbers indicate the best-performing method for each task. | method | BoxInBin (V3) T | BoxInBin (V3) T‑O | BoxInBin $P$ T | BoxInBin $P$ T‑O | Lift (V1) T | Lift (V1) T‑O | Stack (V1) T | Stack (V1) T‑O | Square (V1) T | Square (V1) T‑O | MugHang (V1) T | MugHang (V1) T‑O | Average T | Average T‑O | |-----------|---------------|---------------|--------------|--------------|--------------|--------------|---------------|---------------|---------------|---------------|----------------|----------------|--------------|--------------| | Ours | 0.65 (0.51, 0.77) | 0.04 (−0.01, 0.09) | 0.86 (0.76, 0.96) | 0.02 (−0.02, 0.06) | 0.88 (0.79, 0.97) | 0.26 (0.14, 0.38) | 0.66 (0.53, 0.79) | 0.52 (0.38, 0.66) | 0.68 (0.55, 0.81) | 0.54 (0.40, 0.68) | 0.96 (0.91, 1.01) | 0.82 (0.71, 0.93) | 0.78 (0.73, 0.83) | 0.36 (0.31, 0.41) | | Co-train | 0.44 (0.30, 0.58) | 0.04 (−0.01, 0.09) | 0.76 (0.64, 0.88) | 0.00 (0.00, 0.00) | 0.90 (0.82, 0.98) | 0.14 (0.04, 0.24) | 0.54 (0.40, 0.68) | 0.34 (0.21, 0.47) | 0.66 (0.53, 0.79) | 0.46 (0.32, 0.60) | 0.98 (0.94, 1.02) | 0.72 (0.60, 0.84) | 0.71 (0.66, 0.76) | 0.28 (0.23, 0.33) | | MMD | 0.38 (0.25, 0.51) | 0.00 (0.00, 0.00) | 0.18 (0.07, 0.29) | 0.04 (−0.01, 0.09) | 0.82 (0.71, 0.93) | 0.16 (0.06, 0.26) | 0.44 (0.30, 0.58) | 0.40 (0.26, 0.54) | 0.38 (0.25, 0.51) | 0.34 (0.21, 0.47) | 0.80 (0.69, 0.91) | 0.70 (0.57, 0.83) | 0.50 (0.44, 0.56) | 0.30 (0.24, 0.35) | | T.-only | 0.30 (0.17, 0.43) | 0.00 (0.00, 0.00) | 0.20 (0.09, 0.31) | 0.00 (0.00, 0.00) | 0.82 (0.71, 0.93) | 0.00 (0.00, 0.00) | 0.42 (0.28, 0.56) | 0.00 (0.00, 0.00) | 0.48 (0.34, 0.62) | 0.00 (0.00, 0.00) | 0.64 (0.51, 0.77) | 0.00 (0.00, 0.00) | 0.48 (0.42, 0.54) | 0.00 (0.00, 0.00) | | S.-only | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) | 0.00 (0.00, 0.00) |

Table R1: Sim-to-sim success rates with confidence intervals for image-based policies.

Table R2: Sim-to-sim success rates with confidence intervals for point cloud-based policies.

• Transferring transitions instead of state-action pairs: Our proposed framework does not assume a fixed temporal length for input observations, so extending it to incorporate temporal information—such as full transitions—is straightforward. In this work, we adopt standard behavioral cloning (BC) architectures, namely Diffusion Policy[5] and 3D Diffusion Policy[6], as concrete instantiations to validate the core function of our framework. Exploring extensions that leverage transition-level alignment is a promising direction for future work.
• Inconsistent wording in hypotheses: We will revise the wording of the hypotheses to align more closely with the conclusions presented in the experimental section.
• Missing/incorrect reference: We will correct all reference issues related to tables and figures in the revision.
• Unclear/inappropriate wording: We will carefully review and address any ambiguous or awkward phrasing in the revision.

[1] What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL'21;

[2] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning, NeurIPS'23

[3] What Matters in Learning from Large-Scale Datasets for Robot Manipulation, ICLR'25;

[4] Empirical Analysis of Sim-and-Real Cotraining Of Diffusion Policies For Planar Pushing from Pixels, IROS'25.

[5] Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS'23;

[6] 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations, RSS'24.

Dear reviewer kja7,

We are writing to kindly follow up regarding our response to the reviewer comments for our submission. Please let us know if any further clarification is needed. We appreciate your time and consideration!