ASID: Active Exploration for System Identification in Robotic Manipulation

”Presentation/clarity can be improved: specifically, the abstract and introduction mostly describe the field of active exploration for system identification and adaptive control, as opposed to the specific method proposed, which appears to overstate the paper’s novelty. Furthermore, the approach warrants a better intuitive explanation. As I understand it, the Fisher Information objective attempts to quantify the sensitivity of model parameters to trajectories expected given some policy. Therefore, maximizing this objective yields a policy that, when executed, yields the maximum additional information about the model parameters.”

This is correct. As described in Section 3, the inverse Fisher information matrix serves as a fundamental lower bound on parameter estimation error, so exploring to maximize the Fisher information will collect data that will yield the most accurate estimate of the unknown parameters.

Intuitively, the Fisher information quantifies the sensitivity of the trajectory distribution to the unknown parameter. Note that the Fisher information is defined in terms of the gradient of the log-likelihood with respect to the unknown parameter. Thus, trajectory distributions for which the Fisher information is large correspond to trajectory distributions that are very sensitive to the unknown parameters, i.e., trajectory distributions that are significantly more likely under one set of parameters than another. By exploring to maximize the Fisher information, we, therefore, will collect trajectories that are maximally informative about the unknown parameters, since we will observe trajectories that will be significantly more likely under one set of parameters than another.

”Lack of baselining/adequate discussion of other methods that use the Fisher information objective. The statement “As compared to these works, a primary novelty of our approach is the use of a simulator to learn effective exploration policies” seems too strong and overstated given that there are entire fields dedicated to this, and “the application of our method to modern, real-world robotics tasks” is an inadequate claim to novelty.”

It is certainly the case that using the Fisher information matrix to guide exploration has been proposed before (as our related work makes clear), and, in addition, the use of simulators to aid in real-world robotic learning is also common practice. However, to the best of our knowledge, our work is the first to combine the advantages of using high-fidelity simulators to aid in real-world robotics with active Fisher information-guided exploration for parameter estimation.

In particular, essentially all the works we have cited that rely on the Fisher information matrix to direct exploration (a) do not make use of prior information encoded in simulators to aid in exploration and (b) either do not provide real-world experimental justification for their methods or, if they do, consider only very simple toy problems. In contrast, essentially all the works we are aware of that aim to use simulators to aid in real-world robotic tasks do not perform the type of directed exploration we are proposing to estimate the unknown parameters, but instead use the simulators in a more naive manner, typically by simply transferring a policy that is trained in sim to solves the goal task. Our work can be seen as bringing together these directions—demonstrating that classical works on active parameter estimation can be combined with modern sim2real techniques to solve real-world robotic problems.

If the reviewer is aware of further works that are more relevant to compare to what we have missed in our related work, providing these references would be extremely helpful so we can better situate our work in the existing literature.

”More detail on why the Fisher Information is used vs other methods (observability Grammian, Kalman Filter covariance).”

As described in Section 3, the inverse Fisher information matrix is a fundamental lower bound on the difficulty of parameter estimation and, in addition, serves as an upper bound when the estimator is, for example, the maximum likelihood estimate. Given this, to minimize parameter estimation error (which is the goal of our exploration procedure), we should aim to explore so as to maximize the Fisher information—doing this will collect data that produces the most accurate estimate of the unknown parameters. Thus, in short, maximizing the Fisher information is the mathematically optimal thing to do if the goal is to minimize estimation error. While criteria other than the Fisher information may be used to direct exploration as the reviewer suggests, they lack this optimality property, and would not necessarily collect data that yields the best possible parameter estimate.

We also remark that using the Fisher information to guide the data collection has a long history in the theory of statistics and experiment design. We refer the reviewer to works such as [1] and [2] below for further discussion of this, as well as formal justification of the argument we have presented above.

[1] Luc Pronzato and Andrej Pazman. Design of experiments in nonlinear models. Lecture notes in statistics, 212(1), 2013.

[2] Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.

”It seems that the system relies on the assumption that a learned simulator is able to generate accurate trajectories, but that is not the case for out of distribution trajectories. I understand that exploration precisely minimizes that effect, but the probabilistic model should be able to capture the lack of information in out of distribution data. Currently, the only uncertainty comes from the noise if I understand correctly. ”

We assume the simulator perfectly matches the real environment for some existing setting of unknown parameters (cf. Section 3), i.e., the true environment can be modeled by some setting of the simulator. As such, our estimation problem reduces to estimating this unknown parameter, and we do not consider out-of-distribution trajectories since, if we are able to identify the true parameter, the distribution of trajectories generated by the simulator will match the one generated by the real environment. This assumption underlies a variety of applications of sim2real transfer, which assume that the simulator (with appropriately chosen parameters or distributions of parameters) can effectively model the real world, allowing for policy transfer from sim to real [1, 2].

In cases where this does not hold, training a policy in sim is often still a valuable starting point for learning in real [3]. In such settings, learning a targeted distribution of simulator parameters to train a policy for deployment and finetuning the resulting policy in real often leads to improved performance [3, 4]. While such settings violate our assumptions, our exploration procedure would nonetheless generate informative data to learn such targeted distributions of simulator parameters.

[1] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. ICRA, 2019

[2] Bingjie Tang, Michael A. Lin, Iretiayo Akinola, Ankur Handa, Gaurav S. Sukhatme, Fabio Ramos, Dieter Fox, Yashraj Narang. IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Reality. arXiv preprint arXiv:2305.1711, 2023

[3] Laura Smith, J. Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. ICRA, 2022

[4] Allen Z. Ren, Hongkai Dai, Benjamin Burchfiel, Anirudha Majumdar. AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer. CoRL, 2023

”How did you use RANSAC for tracking?” We use pyRANSAC-3D [1], an open-source implementation for fitting primitive shapes like spheres and cuboids to a pointcloud. The method decomposes a cuboid into 3 plane equations and estimates those by 1) sampling points, 2) fitting a model, and 3) evaluating the model under all available points (c.f. RANSAC). This process is repeated until a threshold criteria or a maximum number of iterations is met. From the plane equations, we can extract the position and orientation of the rod.

[1] https://github.com/leomariga/pyRANSAC-3D/

”While the experimental section is one of the strengths due to the evaluation in a realistic robotic scenario, the methods should also be evaluated on standard benchmarks for comparison, such as HalfCheetah.”

Standard benchmarks like HalfCheetah don’t include varying physics parameters and, therefore, do not cover the tasks we aim to solve. As shown in the follow-up experiments [Appendix Fig. 10d], our method yields directed exploration toward state transitions affected by the changing physics parameters.

”Furthermore, the work of Kumar2019 does not seem to be related to exploration with mutual information as stated in this work.”

The mutual information of a distribution over physics parameters $\Phi$ and corresponding trajectory distribution $\mathrm{T}$ is defined as: $I(\Phi|\mathrm{T}) = H\[\Phi] - H[\Phi|\mathrm{T}]$ with $H[\cdot]$ the entropy. With $H[\Phi]$ a fixed quantity, we can approximate $H[\Phi|\mathrm{T}]$ by learning an estimator $q_{\theta}(\phi|\tau)$ from data parameterized by $\theta$ and predicting the physics parameters $\phi$ from a trajectory $\tau$ (c.f. [1]). Minimizing the prediction error of estimator $q$ over $\theta$ (c.f. [2]) then corresponds to maximizing the mutual information.

”The baseline used [Kumar2019] seems very weak (in Fig 4 it does not explore at all).” The reason [2] performs poorly on our tasks lies in the data used to train the estimator. [2] uses object-centric primitives to collect the data, e.g., the robot always pushing the rod. Every trajectory is, therefore, somewhat informative about the rod's center of mass. Since we assume no access to object-centric primitives, we instead use random exploration to collect the initial data and train the estimator. The data now also contains uninformative trajectories from which the parameters cannot be estimated, e.g., because the robot did not interact with the rod. Due to the long horizon, large state and action space, however, the estimator overfits to all trajectories, being equally certain/uncertain about informative and uninformative trajectories. Now, training a policy using the estimator log-likelihood as a reward leads to the policy learning the initial uninformative random exploration behavior used to collect the data.

[1] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine. Diversity is All You Need: Learning Skills without a Reward Function. CoRL, 2018

[2] K. Niranjan Kumar, Irfan Essa, Sehoon Ha, C. Karen Liu. Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation. 2019

”The main weakness for this paper is that this pipeline is very similar to other exploration methods model-based RL. For example: Shyam et al. Model-based active exploration. ICML, 2019. In fact, the pipeline is quite similar to Shyam et al. albeit the metrics and models used are different. However, due to the similarities in the process, those papers should be discussed and, ideally, included in the comparison.

I do not fully understand the reference to REPS as that is a model-free RL method. There is no transition model estimation. ”

We would like to note that ASID is not quite in the same realm as a model-based RL algorithm. Instead, it considers exploring and collecting data so as to identify the parameters of a physics-based simulator (such as MuJoCo or PyBullet), rather than an end-to-end model as most model-based RL algorithms would do. In the following, we discuss 1) the conceptual difference of ASID from model-based exploration in terms of explorative behavior and model extrapolation, and 2) provide empirical evidence.

1) Conceptual difference of ASID from model-based exploration

ASID fundamentally differs from model-based exploration in the problem setting. Model-based exploration typically leverages fully learned models to guide exploration with the goal of achieving coverage in the state space. The exploration methods proposed by [1] [2] use a learned ensemble of forward models and their disagreement to learn multiple exploration policies that together cover the state space. The collected data is then used to train a downstream task policy. In contrast, our approach relies on the use of an existing physics-based simulator and explores to identify specific unknown parameters in the simulator (e.g. mass, friction coefficients), rather than learning a full model. Our exploration policy searches out state transitions that are most affected by a change in the underlying physics parameters [Appendix Fig. 10c, 10d]. We then roll out a single trajectory in the real world to identify the parameters of the simulator. We use a physics simulator because it extrapolates to unseen states in contrast to a fully learned model [Appendix Fig. 11].

This means we don't learn a dynamics model anywhere in our pipeline and use a model-free black box optimization method (REPS) to identify the physics parameters of our simulator. However, we'd like to note that using a simulator in the process can be seen as a very specific instance of a model-based approach.

2) Empirical Evidence

We construct an experiment to differentiate ASID from Model-based active exploration (MAX) [1]. The environment contains four spheres out of which the red sphere’s friction parameters are varied while keeping the parameters of the three blue spheres fixed. As in our previous experiments, the robot uses delta endeffector control (x,y) and is equipped with a peg instead of a gripper [Appendix Fig. 9].

Investigating the exploration behavior, we show that MAX aims to cover the state space through multiple policies throughout training. Since it uses the disagreement between fully learned dynamics models, it gets distracted by novel states induced by the movement of all spheres (red and blue) [Appendix Fig. 10a, 10b]. In contrast, our method based on the Fisher information yields a single policy that seeks out the sphere affected by the changing physics parameters (red) and ignores the irrelevant spheres (blue) even if they lead to novel states [Appendix Fig. 10c, 10d].

With the collected data of both exploration methods, we train a forward dynamics model $s_{t+1} = f_{theta}(s_t,a_t)$. We model $f_{theta}$ as a three-layer MLP and train using the MSE loss until convergence. When evaluated on out-of-distribution trajectories, i.e., trajectories not included in the training data, we find the model to be extremely inaccurate [Appendix Fig. 11]. While the simulator extrapolates to unseen states and correctly predicts the movement of the sphere, the model hallucinates movement even when the endeffector does not interact with it at all [Appendix Fig. 11c, 11d]! These findings make ASID preferable to a purely model-based approach.

[1] Pranav Shyam, Wojciech Jaśkowski, Faustino Gomez. Model-Based Active Exploration. ICML, 2018

[2] Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta. Self-supervised exploration via disagreement. ICML, 2019

”One aspect that it unclear from the paper is how specific the resulting exploration and task policies are. How generalizable of a policy does the system learn at the end of the day? For example, does the ball pushing policy work only for the specific environment with that breakdown of friction patches and coefficients or is the policy more general and can be used to push balls in a variety of environments? Put differently, do I need to learn a new task policy and calibrate the simulator for every minor varioation of the task description?”

The exploration policy generalizes to different physics parameters of the object of interest that behave similarly [Appendix Fig. 12], e.g., mass, friction, and center of mass can all be identified by pushing the object. Furthermore, the policy shows surprising generalization capabilities to sphere locations unseen during training [Appendix Fig. 12]. Furthermore, [Fig. 4] shows that our policy learns to cover the entire space, allowing it to generalize to changes in, e.g., patch locations.

The task policy generalizes to scenarios seen during training. While we assume a zero-shot transfer of our policy, the approach could be extended by domain randomization, i.e., randomizing over object positions and parameters not identified by our system to train a more robust policy.

”The work mentions that it assumes the optimal policy can be found. That is a rather big assumption for RL as finding the optimum is not guaranteed and the other aspect is that often the reward function does not truly represent what we want to optimize for. Does the proposed approach actually need to find the optimum or is a "good enough" policy also acceptable?”

We do not rely on finding an optimal exploration strategy. To showcase this, we evaluate different training checkpoints of our policy training in [Appendix Fig. 10d]. The results show that even a “good enough” policy that has not converged to the optimal policy is acceptable for running system identification as most of them perform a reasonable exploration strategy i.e., pushing the sphere that is affected by the parameter variations.

”There are simplifying assumptions made for the exploration Fisher loss, how limiting are they?”

The simplifying assumptions on the Fisher information loss are not majorly limiting under standard assumptions on the dynamics. In particular, if dynamics take the form $s_{h+1} = f(s_h,a_h)$, as is the case in the setting we consider, then it is often reasonable to assume that there is some small Gaussian perturbation of the state (this is a common assumption, for example, in much of the control theory literature, e.g. canonical settings such as LQG). Given this, our simplified expression for the Fisher Information in Section 4.1 is in fact not a simplification but is exact. In our experiments, we did not observe that using this approximation resulted in a performance loss, even when training a simulator that does not precisely exhibit Gaussian noise.

“The text states that the system isn't using a differentiable physics engine. If one was used, what would this mean for the method?”

Access to a differentiable physics engine allows one to compute the Fisher information directly. This would make our reward estimate less noisy and stabilize the training of the exploration policy.