Flow-based Domain Randomization for Learning and Sequencing Robotic Skills

Thank you for your detailed review and practical recommendations regarding our claims and experimental metrics, which have significantly strengthened our manuscript. Below we address each of your comments and questions.

… The claim does not hold all the time, for example, in curriculum learning, we tend to pay more time on solving unsolvable regions of the domain in the later learning process. The reference should be attached in the statement or writing should restrict the scope.

We agree that this claim was imprecise, and there are other methods that train adversarially on difficult problems. While this would not be a good strategy in our class of problems due to the infeasibility of some parts of the parameter space. That statement has been updated to say "An alternative strategy is to learn an environment distribution during training with the aim of finding the broadest possible training distribution that can feasibly be solved in order to maximize the chances of transferring to an unknown target environment." Additionally, we added a section to related work discussing adversarial training.

There miss specific metrics or performance indicators about the “outperform” term.

We have modified the statement to a more detailed claim about coverage.

Too broad a sampling distribution and the training focuses on unsolvable environments and falls into local minima. On page3. What is the theoretical support or evidence support for this claim? My understanding about the broad sampling distribution is to cover scenarios as many as possible to achieve DR purpose.

The support for this claim comes from our experimental results and from the results of many other papers on learning for domain randomization. Figure 3 shows that full domain randomization (the broadest possible distribution) is insufficient for reaching the target success threshold. For example, in the “ant” domain where the goal is for the ant to run forward, full domain randomization learns a uniformly applied strategy of bracing the weight of the body so that the negative reward is not received from floor contact. The result is that none of the ants (even the lighter ones) can be considered to have a successful policy because none are running forward.

However, there are some related works [2-4] that deserve discussion or comparison in experiments.

We agree that these are relevant papers, and we have updated our related work to include them.

Throughout the manuscript, I find the coverage ratio works as the primary metric in terms of learning curves and table results. However, its concept is not well introduced at the beginning of evaluation. My understanding about the coverage is related to the reward threshold. Hence, some general metrics can be involved in strengthen this work.

Apologies for any confusion, we moved our discussion of the coverage metric to earlier in the evaluation section (section 5.2) and added detail to how it is calculated in our experiments.

Regarding the other suggested metrics

• Our coverage metric is the same as the success rate metric from Tiboni et al. More specifically, it is the proportion of environments from the entire parameter range that the policy is expected to succeed under. We prefer the name “coverage” because this quantity does not reflect the actual success rate on a robot.
• We agree that this is a good metric to report. In our updated manuscript, we have added a new table with CVaR scores. CVaR was computed as the mean of the final rewards falling below the 10% percentile (VaR).

Thank you for your constructive feedback and suggestions on expanding our related work and improving figure clarity, which have enhanced the overall presentation of our paper. Below we address each of your comments and questions.

The citations can be more comprehensive with more mention of domain adaptation, transfer learning and sim2real transfer. Some examples are mentioned below:

We agree that these are relevant papers, and we have updated our related work to include them.

Also, the authors are requested to expand further on these lines: "Some previous works have combined domain randomization with information gathering via system identification (Ramos et al., 2019; Sagawa & Hino, 2024)"

We have expanded on this discussion further in the related work section. That section of the related work now reads as follows: "Beyond training robust policies in simulation, learned sampling distributions can be tied to the real-world environmental conditions under which policies are likely to succeed. Previous works have integrated domain randomization with real-world interactions for more informed training distributions [citations] or to find the maximally effective real-world strategy [citations]. However, these methods often necessitate expensive policy retraining or data-intensive evolutionary search based on real-world feedback, posing challenges for real-time applications. Instead, we utilize our learned sampling distribution as an out-of-distribution detector within a multi-step planning framework, enabling fast and data-efficient information gathering in the real world."

The use of domain randomization is to result in domain generalization and sim2real. It is difficult to assume that domain randomization alone will achieve this. There could be some more discussion about the other approaches.

We agree that domain randomization alone is not always sufficient for transfer. Hopefully our updated related work discussion helps highlight that real-world feedback is an important component of sim2real transfer.

In terms of the choice of neural sampling distribution, LSDR also uses a learned distribution. Is the main difference that LSDR uses a neural network to fit the GMM parameters? Or do they learn the GMM parameters directly?

The LSDR baseline directly learns multivariate gaussian parameters. They don’t learn a GMM or use neural networks, but rely a simple Gaussian representation.

The writing is unclear in certain places. See below.

Thank you for pointing these out. We have addressed all of these in the updated manuscript. Specifically, we made the following updates:

1. We clarified our statement regarding the usefulness of RL in the introduction. It now reads as follows: "Reinforcement learning (RL) is a powerful tool in robotics because it can be used to learn effective control policies for systems with highly complex dynamics that are difficult to model analytically. Unlike traditional control methods, which rely on precise mathematical models, RL learns directly from simulated or real-world experience [citations]."
2. We updated the caption of figure 5 to much more clearly state what each subfigure is. Please see our rebuttal to Reviewer aKt5 for details.
3. We elaborated on the yaw invariance property of the gear-insertion problem in the main text of the paper. It now reads as follows: "Despite an unknown yaw dimension, the robot is confident in the insertion because the flow $p_\phi$ indicates that success is invariant to the yaw dimension. This is due to the fact that success in the insertion task is defined by the distance between the bottom center of the gear and the base of the gear shaft, which is independent of gear rotation."

Thank you for your thoughtful analysis and positive review. Below we address each of your comments and questions.

For the other mujoco experiments it was not immediately clear how the base sampling distributions were chosen

We attempted to select ranges for these parameters that were large enough to capture all possible physically realistic parameter settings.

I guess one big concern would be how realistic are some of these assumptions made in the paper. In real-world scenarios we often do Sys-ID on the robot to find good initial set of hyperparams and then make it robust around that nominal set. But the proposed approach (in the main paper) uses a very broad initial set which is a bit more complex and clearly a non-learning based approach would not work in this scenario.

We agree with the assessment of how sim-to-real issues are typically addressed, and the limitations of this process are some of the driving motivations for this paper. First, this manual Sys-ID to find a nominal set of hyperparameters requires some human effort on a per-task basis, which we are able to eliminate by selecting broad enough ranges. Second, the nominal parameter values and resulting distributions may not be optimal ones for task performance and robustness. Lastly, the integrated planning system is much more efficient and effective with large parameter ranges as opposed to narrower distributions around nominal values. This is because less information gathering is required by policies that are robust to a larger space of parameters.

Basically, I am curious to hear how would the experiments (mujoco ones) change if the feasible domain regions are not that complex?

If the feasible domain regions are centered and regularly shaped, we see the performance of other learning-based methods increase. The experiments in Appendix A.6 demonstrate this property.

Why was the choice of spline based normalizing flow used to learn the sampling distribution?

We did not have a specific reason to use spline flows over other models outside of its superior performance compared to other architectures in the Zuko library.

Thank you for your detailed insights on experimental design and parameter selection, which have greatly helped us clarify our approach. Below we address each of your comments and questions.

… how are the initial distribution and target distribution decided? How are $\alpha$ and and $\beta$ chosen in Fig. 3?

The target distribution is the same across all methods, as it is a uniform distribution over a set of physical properties such as link masses, joint frictions, and object poses. We attempted to select ranges for these parameters that were large enough to capture all possible physically realistic parameter settings. The initial distribution depends on the method. For GoFlow, the initial distribution is defined by the random initialization of the network. $\alpha$ and $\beta$ were chosen through a hyperparameter selection process detailed in Appendix A.5.

More critically, it was not clear how $J_t$, the threshold for reaching target performance at any environment, is chosen. More discussion/justification (or additional results on varying $J_t$) should be provided.

$J_t$ was chosen to be slightly below the optimal performance under no environment randomization. We verified that the trained policy still exhibited “qualitatively successful” performance at the target reward threshold. For example, we verified that the ant still exhibits running behavior at the chosen target threshold. We have updated the manuscript to describe this selection process. Additionally, we performed an experiment showing how coverage changes with $J_t$. Although we cannot update the manuscript during the rebuttal process, new results in appendix section A.7 show that while the coverage is highly dependent on the choice of $J_t$, GoFlow outperforms baseline methods across almost all choices of $J_t$.

I think the line of work around task-driven system identification [1,2,3] is very relevant

We agree that these are relevant papers, and we have updated our related work [1,2] and introduction [3] to include them.

I recommend putting GoFlow as either the first or the last method in the legend in Fig. 3.

We have moved GoFlow to be the first method in Figure 3.

Do you have intuition why quadruped works the best with \beta=0 as Fig. 7 shows?

While $\beta$ can help with training stability, it can also cause the distribution to converge more slowly. The quadruped domain specifically was less sensitive to large swings in the sampling distribution, and therefore did not benefit from larger $\beta$.

Some of the figures in the paper can be improved. I don't understand the \theta part of Fig. 4. It also took me quite a while to understand Fig. 5, especially top row vs. bottom row. I think the caption can be vastly improved to provide more context and detailed explanations.

Thank you for this suggestion. We have expanded on the captions to figures 4 and 5 to improve clarity. For figure 4, we modified the figure to remove $\theta$ and replace it with yaw, which is how it was described in the caption and elsewhere in the paper. We also defined it in the caption. For Figure 5, we rewrote the entire caption to clearly describe the meaning of each column/row of subfigures. The new caption reads as follows:

"A visual example of the precondition computation described in Section 6.2 for the gear assembly plan shown in Figure 4. The two rows show two different projections of the 3D sampling space (x position vs y position in the top row and y position vs yaw rotation in the bottom row). We apply a threshold $\epsilon$ to the sampling distribution to remove low-probability regions (column 1). Additionally, we filter the value function by retaining only the regions where the expected value exceeds a predetermined threshold $\eta$ (column 2). The intersection of these two regions defines the belief-space precondition, indicating where the policy is likely to succeed (column 3). Comparing the precondition to the beliefs, we can see that the belief is not sufficiently contained within the precondition at $t=0$ (column 4), but passes the success threshold $\eta$ at after closer inspection at $t=4$ (column 5)."