DEAL: Diffusion Evolution Adversarial Learning for Sim-to-Real Transfer

W1 Problem formulation

A1.We appreciate your feedback and would like to clarify the process of our method for you.

The goal of DEAL is to minimize the discrepancy between state transition distributions from different domains as follows:

$$\mathop{\min}\limits_{\theta\in\mathcal{U}}\Vert\mathcal{P}_t(\theta^{target}),\mathcal{P}_s(\theta)\Vert.$$

Our framework is based on GANs, where the generator and discriminator are trained jointly in an adversarial manner. Accordingly, DEAL also aims to minimize the discrepancy through the joint training of generator and discriminator. Notably, our generator DE does not involve neural networks, it generates and optimizes the physical parameter distribution $\theta$ to participate in adversarial training with the discriminator.

Therefore, $\theta$ and the discriminator are optimized jointly, rather than training the discriminator first and then optimizing $\theta$. To improve clarity, we can express the trajectory generated in the simulator as $d^{\mathcal{S}}(\theta,\pi_0)$ and the trajectory pre-collected in the real world as $d^{\mathcal{T}}(\theta^{target},\pi_0)$, then the objective function Equation (6) can be written as follows:

$$\theta^*=\mathop{\arg\min}\limits_{\theta\in\mathcal{U}}\max\limits_{D}\mathbb{E}_ {(\mathbf{s},\mathbf{a},\mathbf{s}')\sim d^{\mathcal{T}}(\theta^{target},\pi_0)}[D(\mathbf{s},\mathbf{a},\mathbf{s}')]-\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}')\sim d^{\mathcal{S}}(\theta,\pi_0)}[D(\mathbf{s},\mathbf{a},\mathbf{s}')] .$$

This formulation makes it clear that the overall objective in Equation (6) describes the joint optimization of $\theta$ and the discriminator, the trajectories from both domains are jointly sampled and input to the discriminator for scoring.

Since we are optimizing over the physical parameter distribution $\theta$ rather than a generator neural network, this may lead to misunderstanding. Thank you for your question and we have refined the problem formulation in the revised version.

W2 Method structure

A2.

1. DEAL follows the same paradigm as GANs, where the discriminator is not pre-trained separately but co-optimized with the generator in an interactive manner. In Equation (5), the source-domain data $d^{\mathcal{S}}$ depends on physical parameters $\theta$ produced by the generator. The inner loop of the discriminator in Algorithm 1 represents epoch-wise training rather than a full pre-training stage. During this interactive process, the source-domain parameters are freshly generated by DE at each step.
2. Equation (6) formalizes the joint adversarial optimization of the discriminator and parameter distribution given by DE, rather than optimizing over a fixed, pre-trained discriminator. In contrast to Equation (4), where the generator learns neural network weights, DEAL performs optimization over a distribution of physical parameters that influence the trajectory, since DE does not involve any neural networks internally.
3. DE plays a dual role as a diffusion-based generator that produces parameters to configure the simulator while simultaneously optimizing these parameters based on the discriminator's feedback.
4. To summarize, DEAL does not include separate discriminator pre-training. Equation (6) defines the overall objective of DEAL. The discriminator's objective is embedded in Equation (6), DE generates and optimizes parameters to adversarially minimize this state-transition distribution gap.

W3 Evaluation

A3.The evaluation metric in Sections 4.1 and 4.3 is the average percentage error of the searched parameters relative to the true parameters, defined as follows:

$$\epsilon_p = \mathbb{E}\left(\left|\frac{{\theta} - \theta^{target}}{\theta^{target}}\right|\right) \times 100\\%$$

Thank you for your suggestion. We have included clear evaluation formulas in the revised version.

W4 Computational cost and data efficiency / Q3

A4.

1. As DEAL does not require a pre-trained discriminator at each iteration. The discriminator and parameter distribution $\theta$ are updated jointly, so while some trajectory data is needed, the amount is not excessive.
2. In Section 4.3, the humanoid task performs sim2sim parameter search using trajectories collected from 200 parallel environments, as mentioned in Experiment Details. Therefore, the real trajectory here refers to 200 = (parallel number = 200)$\times$(trajectory length = 1) state transitions in total.
3. In line 52, this phase is followed by "to train a noise prediction module", indicating it only refers to not using real samples for training the noise predictor. Since DE simulates a denoising process without involving a diffusion process to train the noise predictor.
4. Because DEAL does not separately pretrain the discriminator, the time reported in Table 3 represents the combined training cost of both the parameter distribution and the discriminator.

Additionally, we provide the simulation data budgets for each task, all tasks run 200 parallel environments, but the trajectory lengths differ across tasks. The detailed numbers are shown in the table below.

To compare the data requirements of DEAL and other baselines, we collect between $\eta$=1 and $\eta$=200 trajectories and do parameter search in Humanoid task, then we measure the average search error to evaluate each method's data efficiency.

The table above shows that DEAL achieves the best optimization results using the same amount of trajectory, confirming that DEAL requires less data than other baselines.

W5 Conceptual justification

A5.

1. Due to the non-differentiability of simulators, using least squares or maximum likelihood to compute trajectory gradients doesn't allow backpropagation to physical parameters. Moreover, as the parameter space grows exponentially, gradient-based optimization becomes increasingly intractable. In contrast, DEAL leverages the high-dimensional adaptability of diffusion models to scale more efficiently.
2. We compare DEAL with Model-based method Extended Kalman Filter(EKF) on the CartPole task using the same trajectory data. As shown in the table, EKF suffers from significant bias due to linearization errors when estimating nonlinear parameters. |Avg. Search Error (%)|CartPole| |-|-| |DEAL|6.9±1.1| |EKF|37.4±8.8|
3. Thank you for your suggestion. We have refined and clarified Equation (6).

W6 Minor clarity issues

A6.In DEAL, population individuals iteratively evolve from initial noise, forming a complex distribution that ultimately converges near target parameters. For improved sampling, we fit a Gaussian to the final population's mean and variance, using it as the retraining distribution.

We appreciate your feedback and have addressed other clarity issues in the revised version.

Q1. In Figure 4, how can DEAL outperform the oracle?

A7.When the target parameters are not sufficiently well set, training in the target domain can become relatively more difficult. For example, in tasks like Ant or AllegroHand, excessive mass or weak actuators may slow down oracle training. The searched parameter distribution, due to its bias, may sample easier parameters during training, which can accelerate learning and temporarily result in higher rewards than the oracle. Such fluctuations tend to diminish as training converges, without affecting the oracle’s theoretical optimality.

Q2. Could the authors elaborate on the benefits of their deep-learning-based approach compared to traditional system identification algorithms?

A8.DEAL does not rely on linear assumptions or explicit gradient access often required by traditional methods. Instead, it performs sample-based blackbox optimization, making it broadly applicable to nonlinear systems. Unlike traditional methods that require covariance estimation in high-dimensional spaces, DEAL avoids numerical instability by directly executing denoising updates via Eq. (9). Through discriminator-guided reweighting in Eq. (1), it efficiently identifies high-fitness regions, effectively leading to more targeted and stable updates. The diffusion process further enables adaptive step size control, allowing DEAL to stably and rapidly converge toward the optimal solution.

Q4. What is the architecture of the discriminator network?

A9.In this work, the discriminator is implemented as a fully connected MLP with input $(\mathbf{s},\mathbf{a},\mathbf{s}')$, two hidden layers of 256 units with ReLU activation, and a scalar output. Furthermore, We adopt the WGAN framework to address instability during discriminator training.

Q5. Did the authors consider using active learning during data collection to improve sample efficiency?

A10.We incorporate active learning (AL) for data collection in both Ant and CartPole tasks, one UDR policy is trained with AL by jointly learning an uncertainty predictor to guide exploration toward more informative trajectories, while the other is trained without AL. Each policy is then used to collect a single trajectory of length 200 for parameter search. After retraining new policies with the respective optimized parameters and deploying them in the target domain, we compare their transfer return. As shown in the table below, the results indicate that active learning improves sample efficiency by embedding more target-domain information into trajectories of the same length, thereby enhancing policy adaptability.

Q1. Computational Complexity.

A1. The training cost of the policy itself is not included in our method. Instead, by searching for parameters using DEAL, the need for manual tuning and policy training time is significantly reduced compared to UDR, because after parameter search, the domain knowledge that the policy needs to acquire is of high precision and quality. Moreover, if the new parameters remain within the original bounds, we only need to fine-tune the policy.

Data‑collection cost is also minimal in DEAL. As described in Section 4.4's Go2 Sim2Real experiment, only a single real robot was run for 20 seconds to collect a trajectory containing 1,000 state transitions for parameter search. Furthermore, if DEAL is only used to search for static parameters, such as mass or motor properties, which are intrinsic physical parameters of the robot, the identified parameters can be reused across different controllers and environments, leading to substantial savings in both computational and time costs.

Q2. The method's success hinges on the quality of the initial real-world trajectories used to train the discriminator.

A2. Current learning-based system identification methods do require the collection of trajectories with a certain quality in the real world. However, we believe that even trajectories from very poor policies still encode physical dynamics information. When using the same policy to search parameters, the failed behavior should be reproduced by adjusting those parameters.

To verify this, we train the UDR policies for the CartPole and Humanoid in simulation, and take the policies at various training checkpoints to collect a 200-step trajectory in the target environment. We then search for parameters within $[\tfrac{1}{3}\times\theta^{target},3\times\theta^{target}]$ to reflect the impact of the policies performance on the search results. In the tables, 50 denotes the policy after 50 training iterations, and the policy only starts to converge in the last row.

The tables demonstrate that DEAL is capable of identifying reasonably accurate parameters, even when starting from a poorly performing policy. Naturally, as the controller's performance improves, the quality of the generated trajectories increases significantly, leading to search results that more closely match the ground truth.

Q3. How does the framework handle situations where the real-world dynamics are non-stationary? Could DEAL be adapted for online or continuous system identification?

A3.

1. In non-stationary environments, physical parameters may continuously change, and the collected samples will be multimodal. Since the discriminator can only perform binary classification, it cannot generate effective signals for DE in such cases. However, we believe that future work will develop more general discriminators to match our generator, enabling the accurate identification of non-stationary environment.
2. In online system identification, parameters found by DEAL can directly boost a model-based controller. But for model-free methods like reinforcement learning, fine-tuning in the updated simulator is needed, which isn't real-time. Therefore, all comparisons in this paper are based on offline parameter search. However, for underactuated mobile robots, we usually focus on identifying key internal physical parameters (e.g., joint motor coefficients and mass) that rarely change significantly and are crucial for sim2real transfer. To better adapt to changes in the external environment, we use the RMA controller with an adaptation module to handle environmental variations. This approach complements DEAL's parameter search and significantly improves transfer performance.

Q4. Did you encounter common stability issues in Eq. (6) with UDR?

A4. When training vanilla GANs, we encountered stability issues. WGAN[1] has proved that vanilla GANs minimize the JS divergence between distributions, which becomes constant when distributions have no overlap, causing gradient vanishing and unstable training. To address this, we switched to WGAN, which uses the Wasserstein distance to measure distribution gaps. Unlike JS divergence, Wasserstein distance remains effective even when distributions don't overlap. We also applied gradient penalty or weight clipping to enforce Lipschitz continuity and ensure valid Wasserstein distance computation.

[1] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein generative adversarial networks." International conference on machine learning. PMLR, 2017.

Q1. The clear definition of the simulator parameter theta

A1. In this paper, the specifics of $\theta$ are detailed in Section 4.1 and Appendix A.5. It refers to the physical dynamic parameters of the robot or external environment, such as mass, length and motor coefficients, which directly affect the state transition distribution: $s_{t+1}\sim p(s_{t+1}|s_{t},a_{t};\theta)$. We optimize $\theta$ in the parameter space by using DE to make the corresponding simulated state transition more similar to the real world state transition, and the discriminator is updated jointly in this process, the objective function Equation (6) can be rewritten as follows:

$$\theta^*=\mathop{\arg\min}\limits_{\theta\in\mathcal{U}}\max\limits_{D}\mathbb{E}_ {(\mathbf{s},\mathbf{a},\mathbf{s}')\sim d^{\mathcal{T}}(\theta^{target},\pi_0)}[D(\mathbf{s},\mathbf{a},\mathbf{s}')]-\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}')\sim d^{\mathcal{S}}(\theta,\pi_0)}[D(\mathbf{s},\mathbf{a},\mathbf{s}')],$$

where $d^{\mathcal{S}}(\theta,\pi_0)$ represents the trajectory collected by $\pi_0$ in the simulator parameterized by $\theta$. In this adversarial process, DE acts as the generator to generate and optimize the $\theta$, making the simulation trajectory increasingly similar to the real world trajectory $d^{\mathcal{T}}(\theta^{target},\pi_0)$ from the discriminator's perspective, while the discriminator aims to distinguish between them as much as possible.

Thank you for raising this issue. We have added a clear definition of $\theta$ in Section 3 of the revised version to facilitate your understanding.

Q2. What is the formula for simulator function that theta is optimizng or apart of?

A2. $\theta$ refers to the physical parameters of the robot or external environment. It is not the weight of neural networks or a parameter of function. The formula for simulator optimization related to $\theta$ is Equation (6) in answer A1. In DEAL, we use DE to denoise and optimize $\theta$ to minimize the discrepancy between the real and simulated state transition distributions as perceived by the discriminator.

Q3. In the GAN formula in equation 6, what variables in the equation depend on theta?

A3. In Equation (6) from answer A1, the embedded GAN objective function includes simulated trajectories $d^{\mathcal{S}}(\theta,\pi_0)$, which depend on $\theta$. We optimize $\theta$ using the generator DE and thereby parameterize the simulator, which in turn affects the trajectories collected from the simulator. These trajectories are then used in adversarial training against the discriminator, since the discriminator is always trying to distinguish between simulated and real-world trajectories.

Q4. How do you decouple the policies being different from the dynamics being different?

A4. We use the same policy when collecting real-world trajectories and when searching for parameters or collecting simulated trajectories. This ensures that the parameters we search for are solely aimed at matching the dynamics of the real-world environment.

Specifically, we first train an initial sub-optimal policy $\pi_0$ in simulation based on UDR, and then deploy it in the real environment to collect offline trajectories $d^{\mathcal{T}}(\theta^{target},\pi_0)$. During the subsequent parameter search, we continue to use $\pi_0$ to collect trajectories from the simulator parameterized by $\theta$, denoted as $d^{\mathcal{S}}(\theta,\pi_0)$. The discriminator then samples from both datasets and assigns scores to distinguish them, updating its network accordingly, while DE updates the distribution of $\theta$ based on the discriminator's evaluation.

Q5. In DEAL, are you jointly learning a policy, simulator parameters, and a discriminator?

A5. No, these three components are not trained jointly. The sub-optimal policy $\pi_0$ is pre-trained via UDR, and its weights are frozen during the parameter search as shown in Figure. 1. Therefore, only the simulator parameters $\theta$ and the discriminator are updated during the search process. This procedure eventually converges to a set of physical parameters that approximate the real-world dynamics, which are then used to reconfigure the simulator and retrain a new policy which better adapted to the real world.

Q6. What are the parameters in the simulators you're trying to learn? How do you decide on these parameters to learn?

A6. The parameters contained in $\theta$ are listed in Appendix A.5. These mainly include the parameters that were randomized during the UDR training process, such as the joint PD gains for the Go2 robot or the pole length and mass in the cartpole task. These are key factors affecting sim-to-real transfer, and our goal is to identify more accurate distributions for them, in order to reduce the computational cost and performance degradation introduced by UDR.

Q1.The idea of computing a similarity measure based on collected rollouts on source and domain to optimize a GAN-style process is not new (eg SimGAN).

A1. We appreciate the inspiration provided by SimGAN. DEAL introduces several key modifications to address known issues in GAN-style system identification.

1. Prior approaches focused solely on comparing the similarity of simulator rollouts to real-world demonstrations, with the single goal of making simulated trajectories mirror real-world demonstrations as closely as possible. In contrast, our approach shifts the focus: we employ a discriminator to assess the similarity of physical state transitions, aiming to fundamentally bridge the gap between simulation and reality. This not only enables more efficient use of real-world data but also leads to improved performance.

2. Previous methods (such as SimGAN) adopt the traditional GAN architecture, using a neural network generator to produce simulator parameters for optimization. However, GAN training is highly unstable, and training the neural generator is time-consuming, requiring significant effort in tuning and debugging. In our work, we introduce Diffusion Evolution (DE) as an alternative for parameter search, which significantly accelerates simulator parameter optimization. Moreover, the combination of DE and the discriminator offers high training stability, avoiding the instability issues commonly encountered in traditional GAN-based methods.

Q2. Is there any theoretical justification for using DE in terms of convergence or sample efficiency ?

A2. GAN-style parameter search can be viewed as a black-box optimization problem. In DEAL, we choose DE as the generator mainly due to its convergence stability and efficiency, making it more suitable for scaling to complex environments.

Previous black-box optimization methods such as evolution strategies (ES) typically model the parameter distribution as a Gaussian, updating its mean and covariance in each sample iteration. However, the new samples are randomly drawn from the updated distribution and lack explicit control over the convergence trajectory, often resulting in slow convergence and low final accuracy. Instead of blindly perturbing parameters in each iteration, DE performs a re-weighted posterior estimation of the optimum based on observed samples. This allows the search process to focus more quickly on regions close to real data distribution.

Specifically, DEAL updates the parameter distribution using the following formulation Eq. (9):

$$\theta_{t-1} \gets \sqrt{\alpha_{t-1}} \hat \theta_0 + \sqrt{1-\alpha_{t-1}-\hat\sigma_t^2} \hat\epsilon +\hat\sigma_{t} \omega ,$$

where the estimated optimal parameter $\hat\theta_0$ is given by Eq. (1):

$$\hat \theta_0(\theta_t,\alpha,t)=\frac{1}{Z}\sum_{\theta \in \Theta_t} \underbrace{g[f(\theta)]} _ {p(\theta_0 = \theta)} \underbrace{\mathcal N(\theta_t; \sqrt{\alpha_t}\theta, 1-\alpha_t)} _ {p(\theta_t | \theta_0 = \theta)}\theta , \quad g(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}.$$

In Eq. (1),  $p(\theta_t \mid \theta_0 = \theta)$ is defined by the forward diffusion process: $\theta_t = \sqrt{\alpha_t}\theta_0 + \sqrt{1-\alpha_t}\epsilon$, it acts as a neighborhood weight around parameter $\theta_t$. $p(\theta_0 = \theta)$ represents a probability derived by applying a softmax function $g(\cdot)$ to the discriminator's output, reflecting the likelihood that a state transition under parameter $\theta$ resembles one from the real environment. This formulation effectively uses a softmax-style weighting to emphasize high-fitness regions, guiding $\hat\theta_0$ rapidly toward the true optimal parameters. As a result, the estimate $\hat\theta_0$ serves as a proxy for the gradient direction of the WGAN's objective function $\mathcal{L}$ in Eq. (5) as follows, then the population is updated directionally via Eq. (9) to move toward this optimal estimate.

$$\underbrace{\mathcal{L}}_{\searrow} = \mathbb{E} _ {(\mathbf{s},\mathbf{a},\mathbf{s}')\sim d^{\mathcal{T}}(\theta^{target},\pi_0)}[D(\mathbf{s},\mathbf{a},\mathbf{s}')]-\underbrace{\mathbb{E} _ {(\mathbf{s},\mathbf{a},\mathbf{s}')\sim d^{\mathcal{S}}(\theta,\pi_0)}[D(\mathbf{s},\mathbf{a},\mathbf{s}')]} _{\nearrow}.$$

The diffusion noise schedule also helps to stablize the convergence by determining the update step at each iteration. As $t \to 0$, $\alpha_t \to 1$ and $\sigma_t \to 0$, reducing the variance of $\mathcal{N}(\theta_t; \sqrt{\alpha_t} \theta, 1 - \alpha_t)$ and encouraging fine-grained, local updates in the later stages. The update $\theta_{t-1} \to \hat\theta_0$ serves as a denoising step, similar to the reverse process in diffusion models. With a well-trained discriminator, this process enables stable convergence toward the true optimum.

Additionally, unlike traditional ES that rely on elitism or truncation, which can lose global information and lead to premature convergence to local optima, DEAL utilizes the entire population in Eq. (1). It assigns soft weights based on discriminator-derived fitness to estimate the gradient direction of the WGAN's objective function. This leads to more targeted and stable updates, offering improved convergence stability, efficiency and accuracy over previous methods.

We appreciate your insightful question regarding the theoretical motivation behind DE. We have incorporated this detailed analysis into the revision to strengthen the conceptual grounding of our method.

Q3. The detailed description of the discriminator? How is it regularized to avoid overfitting ?

A3. The discriminator used in DEAL adopts a fully connected MLP with input $(\mathbf{s},\mathbf{a},\mathbf{s}')$, two hidden layers of 256 units with ReLU activation, and a scalar output. In DEAL, we employs WGAN [1] , which utilizes Wasserstein distance to measure the distribution discrepancy. When two distributions do not overlap, the Wasserstein distance can still accurately reflect their proximity. Furthermore, we apply gradient penalty or weight clipping to regularize the discriminator, preventing overfitting or model collapse.

Thank you for pointing out this issue. We have included the detailed description of the discriminator in the revised version.

[1] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein generative adversarial networks." International conference on machine learning. PMLR, 2017.

Q4. Are there cases or scenarios where DEAL underperforms ?

A4. In current tasks ranging from low-dimensional to high-dimensional scenarios, DEAL significantly outperforms other baselines. However, DEAL also shares some common issues faced by GAN-style methods.

When the target distribution is highly multimodal, existing GAN-style methods may struggle to discriminate among multiple source and target distributions at once, as noted in Appendix A.4 (Limitations). Although diffusion models can capture multimodality, the discriminator still averages over diverse state transitions rather than converging to distinct solutions. In this case, DEAL tends to homogenize the influence of such multi-source state transitions rather than converge to multiple exact solutions. However, we believe that future work will develop more general and powerful discriminators to match our generator, enabling the simultaneous identification of multiple environment parameter sets.

Q5. How would modify your approach to reduce reliance on real-world rollouts ? What happens if only a fixed set of real-world data is available ? Which is often the case in practice.

A5. It should be clarified that DEAL uses only offline trajectories. As shown in Alg. 1 and Figure. 1, the policy is not run simultaneously in the simulator and in the real environment during parameter search. In other words, DEAL itself uses a fixed set of real-world data. Specifically, we first collect fixed length offline trajectories in the real world using the UDR policy. During parameter search, we only collect trajectories in simulation and sample from the stored real-world trajectories, using the discriminator to distinguish them until their distributions align.

To further reduce the reliance on real data, we introduce an active learning (AL) module. During UDR policy training, we jointly train an uncertainty predictor, which biases policy actions toward more informative trajectories. In the Ant and CartPole task, we collect one real‑world trajectory of length 200 with each of the standard and active learning models. After conducting parameter search for each, we retrain the new policies and evaluate their returns in the target domain. The table shows the result, which demonstrates that active learning indeed improves sample efficiency by embedding more target‑domain information into trajectories of the same length, thereby improving transfer performance.