Generation of Geodesics with Actor-Critic Reinforcement Learning to Predict Midpoints

Thank you for the review!

This is more of a clarification of the problem setting. It seems that the end goal of the learned policy is not necessarily finding the shortest path. The success criterion is stated as “all values of C(4) for two consecutive points are not greater than $\epsilon$”, which does not imply that a path is the shortest. Is this correct? If so, this should be stated more clearly.

Indeed, judged as a success does not mean being the shortest, but success rates should increase if shorter paths are chosen for all pairs. The reason we do not use the lengths of paths for evaluation is that, since the metric can only be calculated locally, lengths cannot be calculated when the success condition is satisfied. Moreover, it is needed to compare fairly with the sequential baseline method, which has a different generation method. For these reasons, we chose this evaluation scheme.

Please provide some motivations for the definition of $C$ (Equation (4)). Also please explain $df_x$ what is in this definition.

Sorry for the missing explanation. $df_x$ is the differential (pushfoward) of $f$ at $x$.

Since the Finsler distance is defined by (1) and the geodesics can be approximated by the segment in the coordinate space, we approximate the distance between the closed points by (5). Furthermore, since $F$ is continuous and the points are closed, this integral can be approximated by the value at $t=0$. This is (4).

Why is Equation (5) hard to compute efficiently?

Of course, there are various methods of numerical calculation of integrals, but they take more time than calculating a function once and it is difficult to know in advance the required accuracy.

In Equation (11), should the right-hand side be $d(x,y)$, the true distance rather than the local approximation? Either way, Equation (11) could use a more expanded explanation.

Unlike the proposed method, different policies and value functions are considered here for different depths. $V_i$ represents the sums of the values of $C$ after the policies $\pi_{i-1}, \ldots ,\pi_0$ is applied. It is therefore correct that the right-hand side of (11) is $C$.

In Proposition 2, what is $V_i$ ?

This is a typo of $V_i^*$. Thank you for pointing this out.

In the Sequential Reinforcement Learning (Seq) baseline, why is the reward function (Equation (16)) scaled by $\epsilon$ ? How does this decision affect the learning?

This motivates agents to reach goals in as few steps as possible.

The first concern that the experimental environments were artificial could have been answered by conducting an additional experiment.

One of the reasons we focused the baselines on reinforcement learning is that it is difficult to make a fair comparison with searching methods. While searching methods can find good solutions if enough time is spent, prediction by trained policy uses very little time. Therefore, we decided to limit ourselves to RL methods for situations where we want to generate paths or motions in a short time.

Thank you for the review!

Perhaps the motivation of this work can be better written. As the authors pointed out in their experiments, generating geodesic (path planning) can be simply tackled by RL by specifying a reward function related to the difference in distance. But it may have instability or other issue compared to path planning approaches.

We are grateful for your useful comments. Indeed, we felt that sequential RL is unstable for both random seed and hyperparameters compared to the proposed methods. It may be due to the fact that sequential RL have long horizons compared to the proposed method.

Could you give some explanation why Car-like task favors your approach, while Matsumoto task not?

I think that sequential RL is not good at CarLike rather than our method is good at CarLike. Since sequential RL uses ad-hoc reward function defined in (16), there is no guarantee that it works well. Perhaps the defined reward is far from the actual distance in CarLike.

Minor: The description of methods in the experiments can be more complete – add a line of “ours” using Eq. 8 before “the following variants of our methods”. The name “sequential RL” is a bit confusing as RL is sequential in nature. Perhaps “vanilla RL” or just “RL”, because your approach uses a non-conventional actor loss.

We will add a little explanation of what "Ours" means.

The name "sequential" may be somewhat inappropriate, but I wouldn't dare change it because previous work by Jurgenson et al. have used this word.

Thanks for the review!

Can we add some more explanation and justification on why the midpoint is not just a trivial extension of the existing method using arbitrary waypoints?

We think that he most non-trivial point for midpoint generation compared to the sub-goal tree generation is the theoretical soundness proved in the subsection 4.3. While the existing method using arbitrary waypoints can perform badly in our setting as in the situation of Remark 6, our midpoint generation method avoids this.

Can we add more analysis on how the proposed method could be generalized to environments with obstacles?

Please see the papers cited in our response to Reviewer etxs.

In Figures 2 and 3, the proposed method does not show clear improvements compared to baselines. Is this expected? Can we add more explanations?

In CarLike environments, the proposed method clearly outperformed other methods except the intermediate point variant of the proposed method. In Matsumoto environments, the proposed method outperformed other methods except the sequential RL baseline. Indeed, the superiority of the proposed method over the intermediate point variant may not be so obvious. However, it is noted that the intermediate point variant can perform badly in Remark 6. Thus, our method has been shown to be theoretically or experimentally superior to all other methods.

Thanks for the review!

Can the authors provide additional examples where their method might be applicable, specifically within the realm of robotics tasks like locomotion and manipulation?

Some robotics tasks including manipulation are formulated by Riemannian geometry in the context of Riemannian Motion Optimization. See the following papers:

Ratliff, Nathan, Marc Toussaint, and Stefan Schaal. "Understanding the geometry of workspace obstacles in motion optimization." 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015.

Mainprice, Jim, Nathan Ratliff, and Stefan Schaal. "Warping the workspace geometry with electric potentials for motion optimization of manipulation tasks." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.

Therefore, our method is potentially appliable to these tasks. Especially, since the technique to warp metrics for avoiding obstacles is already studied, our method can be appliable to environments with obstacles.

The sentence "only works well locally since we assume that manifolds have global coordinate systems and the continuous midpoint property may be satisfied only locally" we wrote is maybe misleading. We assumed the continuous midpoint property for the theoretical result, it seems not to be practically necessary. Indeed, even environments we used for experiments seem not to have the property. Therefore, this assumption is a limitation of our theoretical contribution and not of our method.

How does the proposed approach compare in terms of benefits and applicability to other realistic tasks, beyond what has been demonstrated in the paper?

In the context of aforementioned Riemannian motion optimization, the practical advantage of RL methods compared to the optimization method is the computational cost. While optimization is computationally costly, predictions by learned agents are fast. The advantage of our method over sequential RL method is that there is no need to reward engineering. In addition, the fact that the proposed method has a very short "horizon" compared to sequential RL may stabilize learning.

Could the authors discuss the relationship and distinctions between their work and recent research on quasimetric learning for goal-conditioned RL?

Thank you for sharing related works! The main difference between our baseline sequential RL and the quasimetric learning method is architectures of value function approximation models. Since we do not focus on architectures of critics and used simple MLPs for both the proposed method and the baseline methods, the direction of our research and the direction of this research are orthogonal. The embedding method for quasimetric learning can be used for the critic in our method.