Flow-based Maximum Entropy Domain Randomization for Multi-step Assembly

Thank you for your careful review and for acknowledging the novelty of combining normalizing flows with maximum entropy principles. We hope our responses below, along with the updated manuscript that includes enhanced real-world experimental results, address your concerns.

For the comparisons against baseline methods, the range of tasks is too narrow (only 4 simulation tasks) … / … If the aim is to make domain randomization easier or better for RL, the experimental results need to demonstrate that on a wider range of tasks, such as tasks that are highly relevant to real robotic systems, such as quadrupeds / humanoid locomotion tasks, manipulation tasks, etc, or simulation benchmarks that contains more than 4 tasks.

The initial version of the paper had 5 simulation tasks which include, as suggested, quadrupeds (Anymal domain) and manipulation (Gears domain). Humanoid locomotion was not in the initial task set, but we have now run additional analysis on a humanoid domain as well, bringing the total number of simulated domains to 6. This is comparable to the number of tasks in all of the cited baselines such as DORAEMON (6 domains), LSDR (2 domains), ADR (1 domain).

Out of the 4 sim tasks, it is difficult to assess the effectiveness or significance of GoFlow's improvements or better performance, especially when all methods and tasks have relatively low success rates. For instance, in quadcopter, it is hard to draw a conclusion that 0.02 achieved by GoFlow is indeed better than baseline methods in a statistically significant manner.

The reason that the success rate is low in some domains is because we intentionally expanded the number and range of domain randomization parameters such that only a small subset of those ranges was feasible. This was done for two reasons. First, we aim to show how GoFlow performs better than baselines in such domains where the feasible parameter space is a small, off-center, and irregularly shaped distribution within the global sampling distribution. Second, small feasible parameter spaces illustrate the need for a planning system that can gather information to ensure that the agent is within the small preimage of a successful skill.

While the success rate under the global sampling distribution is small in some tasks, this does not have any bearing on the statistical significance of GoFlow’s performance against baselines. To ensure this, we performed statistical tests, which can be found in Appendix A.6.

Thank you! To answer your question, we perturb the end-effector pose by a random $\pm 0.01$ meter translational offset along the x dimension during the pick. We expect some additional grasp pose noise due to control error and object shift during grasp. These details are now noted in the paper as well.

We appreciate your detailed feedback and for highlighting the potential of our method to extend state-of-the-art domain randomization methods while highlighting key areas for writing improvements and deeper analysis.

The authors present the method as a novel approach for learning DR distributions. However, to my understanding, the presented opt. problem in Eq. 5 has been recently proposed by Tiboni et al. [1] (aka DORAEMON) under an equivalent formulation, which, in turn, took inspiration from the self-paced curriculum learning opt. problem [2] […] However, the authors fail to make connections to these existing works, while proposing an equivalent formulation. The authors shall explain which parts of the method are novel, and which come from existing works in literature.

Thank you for pointing this out. We agree that Section 4 would benefit from more explicit citations to these relevant works that have inspired our formulation. The optimization problem presented in Eq. 5 is indeed aligned with these prior methods and is not a novel contribution of our work. Our main novelty is a more flexible and expressive neural sampling distribution and showing how that distribution can be integrated into a multi-step planning framework.

We have revised Section 4 to acknowledge these works explicitly and clarify the scope of our contributions. Specifically, we have cited Tiboni et al. [1] and self-paced curriculum learning [2] and discussed how our approach builds on these ideas. The updated section also delineates the aspects of our method that are novel versus those that are extensions or adaptations of existing literature.

As a result, the algorithmic implementation in Algorithm 1 appears disconnected from the motivation and the general problem described by Eq. 5.

We agree that the explanation of Algorithm 1 was not sufficiently connected to the optimization problem defined in Eq. 5. We have elaborated on this connection in the updated manuscript.

The final objective for optimizing the sampling distribution in line 9 of Algorithm 1 includes a third term beyond the entropy and the KL term which is not discussed in the main text. It's also unclear why this term is added.

The first loss term in line 9 of Algorithm 1 is the gradient of the expected return with respect to the parameters of the normalizing flow, which pushes the sampling distribution to maximize reward. This expression is also used in [2], and it is related to the expression used in policy gradient methods with respect to the policy parameters. We have updated the paper to better explain this and cite relevant sources.

[…] Perhaps more importantly, this term was introduced by LSDR [3] but no connection/citation is reported in the manuscript by the authors. In fact, the final algorithmic implementation in Algorithm 1 appears to be equivalent to LSDR, but extended to normalizing flows.

Yes, the extension of the LSDR-style of learned domain randomization from multivariate gaussians to normalizing flows was one of our novel contributions. This representational choice proves to be very important in domains with irregular and uncentered feasible sampling regions. Our second novel contribution is demonstrating how learned control policies can be integrated into a belief-space planning framework that enables information gathering prior to policy execution.

DORAEMON [1] recently showcased successful deployment of FullDR, ADR, and DORAEMON in the cartpole and other locomotion tasks. However, in this paper these methods seem to score critically low in the same environments. This raises questions on the validity of the experimental evaluation, if not properly analyzed and discussed. E.g. why are the success rates so low across all methods?

The exact empirical results from other papers such as ADR, DORAEMON, and LSDR are not directly comparable. In addition to testing on different simulators, the papers deviate significantly in their domain randomization ranges. Focusing on DORAEMON, the randomization ranges were selected to be within some window of a known-successful center point. In contrast, our ranges are much larger and not necessarily centered. To show the relationship between domain ranges and success rate, we conducted an additional experiment comparing the range size to the success rate (Section A.3). From this experiment, you can see that GoFlow better handles larger domain sizes while still matching success rates like those reported in prior works for smaller ranges.

Further, low success rates imply that the policy is only feasible in a small sliver of the original domain ranges. The motivation behind multi-step planning section (section 6) was to show that if this sliver (or precondition) is identified, we can plan to move inside it, resulting in a policy with high likelihood of success.

Thank you for your response. Your feedback has been instrumental to improving the paper quality. Below we address your updated list of concerns.

I believe it could be discussed more thoroughly in the paper for a cleaner theoretical explanation and interpretation.

Thanks for this suggestion. We agree that it would be beneficial to show a derivation of this loss term. We have added a section to the appendix deriving it (Section A.7), and referenced that from the main text where the expression is used (Section 4).

the paper could benefit from dedicating a bit more time into making clearer connections of GoFlow to the closely related DORAEMON, LSDR and self-paced papers

To make this connection clearer, we have added another paragraph to the method section after the optimization problem has been defined to more clearly outline how other related works formulate and solve this problem. If this is not what was intended, could the reviewer please elaborate on what additional information would make this connection clearer?

I believe the authors should also discuss the tuning process for the hyperparameter alpha

The tuning process is discussed in the Appendix (Section A.2). We performed parameter sweeps for all baseline methods, including DORAEMON which doesn’t have the exact same hyperparameters, but instead has other relevant hyperparameters that need to be tuned to trade off entropy and reward.

unfair baseline comparison: the novel Fig. 14 importantly sheds light on the behavior of the DORAEMON baseline, which is never really able to increase its own entropy. This is likely due to the violated assumption of the method, i.e. policy training should start on a feasible, fixed point in the parameter dynamic space.

We acknowledge that DORAEMON performs effectively in domains where its assumptions hold, specifically that training starts at a feasible point in parameter space and that dimensions of variability can be characterized independently. However, these conditions may not always hold (See Figure 2). One of the primary goals of our work was to develop a method that does not rely on such assumptions.

However, we agree that this was not sufficiently conveyed in the initial draft. To address this, we now discuss in paragraph 3 of Section 5.2, as well as in the caption of Figure 2, that the test problems we use explicitly violate some of the assumptions underlying DORAEMON and other methods. Our updated draft also contains experiments in Section A.3 showing how the baselines degrade more quickly than GoFlow with increasingly wide and off-centered ranges.

Finally, regarding ADR, our implementation initializes the starting range to match the size of the interval increment, resulting in higher entropy at the beginning of training.

Thank you for your thoughtful review and for recognizing the importance of improving control policy robustness, as well as the significance of our experimental validation in a challenging gear assembly task. Below we address each of the reviewer's concerns and questions.

The main novelty of using normalizing flows to learn the parameter sampling distribution is somewhat limited. However, it has been shown to be quite effective.

We believe the novelty of this work is two-fold. First, prior work on learning domain randomization distributions made the assumption that the learnable parameter distribution is centered in the parameter range and conforms to a standard parameterized probability distribution. Here, we remove those assumptions by using a more expressive representation (normalizing flows). Second, we demonstrate how learned control policies can be integrated into an uncertainty-aware planning framework that enables information gathering prior to policy execution.

Error in estimating the belief space precondition can lead to incompleteness in planning.

We agree with the reviewer’s statement that determinized probabilistic planners are incomplete. However, this does not limit their usefulness, and often superiority over, complete planners in many problems [1]. While our planner was primarily designed as a proof-of-concept for how to integrate learned preconditions into larger decision making framework, extensions to more advanced, and potentially complete, belief-space planners is an interesting direction for future work.

What is the true uncertainty distribution in the gear assembly task? Is it artificially induced or due to to uncertainty in perception and actuation? How is it simulated during training?

In the gear assembly task, the uncertainty is in the relative pose between the gripper and the gear object. During simulation, the gear is placed in the robot’s gripper with a random offset. Because the gear pose is not in the observation space, the robot must design a conformant policy that handles arbitrary offsets.

How sensitive is the method to the regularization coefficient hyperparameter? The scale of the expected return J depends on the reward function. Does one need to tune the coefficient for every reward function?

Please see the appendix for details on our hyperparameter search. We find that sensitivity to these parameters varies per task. Empirically, we found that a similar range of alpha/beta values tended to work best across tasks.

As a side note, we performed hyperparameter search for all baselines according to their respective hyperparameters. Our results report the best-performing hyperparameter set per method.

The final objective for optimizing the sampling distribution in line 9 of Algorithm 1 includes a third term beyond the entropy and the KL term which is not discussed in the main text.

The first loss term in line 9 of Algorithm 1 is the gradient of the expected return with respect to the parameters of the normalizing flow, which pushes the sampling distribution to maximize reward. Because we are taking the gradient, we can equivalently use the log probability instead of the probability directly, which leads to better numerical stability. This trick is also used in [2] and in policy gradient RL, but with respect to the policy parameters. We have updated the paper to better explain this and cite relevant sources.

[1] Yoon, S., Fern, A., & Givan, R. (2007). FF-Replan: A Baseline for Probabilistic Planning. Proceedings of the AAAI Conference on Artificial Intelligence.

[2] Mozifian, M., Gamboa Higuera, J. C., Meger, D., & Dudek, G. (2020). Learning Domain Randomization Distributions for Training Robust Locomotion Policies. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)