Harmonized Learning with Concurrent Arbitration: A Brain-inspired Motion Planning Approach

7. Why not accommodate more strategies

(1) more strategies

Reliability inference and harmonization in the inference buffer can be time-consuming and computationally expensive. If satisfactory performance can be attained with a small number of strategies, there is no need to include additional strategies. This is particularly relevant in practical scenarios where agents must make rapid decisions.

(2) more learnable policy/value networks

Thanks for your suggestion. Accommodating more learnable policy/value networks and switching between them is very valuable and we will study it in the future. The main reason that we only include a pair of policy/value networks in this experiment is as follows. The initial performance of learning-based algorithms is relatively poor and thus learning data needs to be collected by using non-learning strategies. The model gradually is self-improved upon its prior successful experience.

8. When to conduct a reliability assessment

We perform reliability inference for each planning problem, instead of before expanding states.

9. Concerns about new probe strategies

(1) Brain model

In the human decision-making reasoning model, $\omega$ denotes a human behavioral strategy. When $\omega_{k}$ proves unreliable while other strategies in the inference buffer $\mathcal{I}$ remain reliable, the strategy with the highest reliability in $\mathcal{I}$ is selected. If none of the strategies in the inference buffer are unreliable, a new probe strategy $\omega_{p}$ is randomly developed from long-term memory. Notably, $\omega_{p}$ is not trained from the initial $\omega_{k}$ initialization.

(2) Algorithm

During the initialization phase, $\boldsymbol{\Omega}$ = {$\omega_{1},\omega_{2}, \omega_{3},\omega_{u}$} and $\mathcal{I}$ ={$\omega_{1},\omega_{2}, \omega_{u}$} were designed to imitate long-term memory and inference buffer, respectively, where $\omega_{1},\omega_{2}, \omega_{3}$ are existing strategies, the network structure of $\omega_{u}$ is existing. Similarly, If strategies in $\mathcal{I}$ are reliable, the strategy with the highest reliability is selected. In contrast, $\omega_{3}$ serves as a tentative probe strategy to guide agents in planning a path. This path is added to the training buffer as before. Moreover, if the strategies in $\mathcal{I}$ exhibit unreliability consecutively twice, $\omega_{3}$ replaces the strategy with the minimum reliability. Importantly, even in the scenario where $\omega_{u}$ is replaced, i.e., absent from the inference buffer, the samples collected persist in training $\omega_{u}$.

(3) Experiment

Observations from the experiments highlight that: two consecutive strategies in the inference buffer remain unreliable is a rare phenomenon. There are three reasons for this happen. (i) Our selected strategy demonstrates greater robustness. (ii) The threshold for reliability is not quite stringent. (iii) It is worth considering that the environments may not be quite complex.

For the third reason, we validated all methods by generating environments with larger dimensions and a higher density of obstacles. Experiment details can be found in Appendix D of the updated paper.

Thanks for your careful review and important suggestions. We have revised manuscript accordingly to make it clearer. We address the specific questions as follows.

1. Parameters

The values for the reliability threshold $\alpha$ and the perceived volatility parameter $\tau$ lie within the range of $[0, 1]$. Theoretically, a larger threshold $\alpha$ leads to better results. However, setting a higher threshold will result in longer experiment runtime, with marginal or possibly no performance improvement. We adopt non-stringent thresholds, with $\alpha$ specifically set to 0.7.

Given that different strategies are adaptable for dealing with distinct environments, we introduce the parameter $\tau$ to represent the preference for strategies in the inference buffer. In our experiments, we assume an equal likelihood for the selection of all strategies. Therefore, $\tau$ is set as 0.5. Besides, we provide details about the hyperparameters of the algorithm in Appendix C of the updated manuscript.

2. Optimization Problems

Sorry for the inappropriate expression. We can't guarantee that $V(s;\boldsymbol{\theta})$ is an overall optimal policy. $V(s;\boldsymbol{\theta})$ can be improved based on successful experiences generated by historical local optimal strategies. Therefore, we have changed it to “$V(s;\boldsymbol{\theta})$ and $\pi(s'|s;\boldsymbol{\theta})$ gradually learn from previous successes and improve themselves as they encounter more problems and better solutions.” in the updated version.

3. Typos in writing

We have improved the listed typos in the updated manuscript.

4. Resources and time complexity

We calculated Params for learning-based algorithms and demonstrated the planning runtime for all algorithms to find a successful path.

We demonstrated the total running time for 1000 test problems across all algorithms to find a successful path. A snapshot of the results is as follows. The complete table and discussion are updated in Appendix A of the revised manuscript. To be more clearly represented in the figure, the results are normalized based on the running time of RRT*. HLCA, GNN+smoother, GNN, and NEXT are learning-based algorithms, others are non-learning algorithms.

A common concern about learning-based algorithms is that their time cost due to the frequent calling of a large neural network model at the inference phase, such as NEXT. The running time is higher than non-learning algorithms because our algorithm is improved based on NEXT and runs multiple candidate strategies concurrently. Nevertheless, our algorithm reduces the running time compared with NEXT. Future work could focus on using pre-computation or caching, e.g., reusing past results to overcome some of this overhead.

5. Pseudocode of the algorithm

We have provided the pseudocodes of all our algorithms in Appendix E of the updated manuscript.

6. Doubts about manually designed strategies

There might be some misunderstandings about our harmonized learning settings. It is the strategy set $\boldsymbol{\Omega}$ and the inference buffer $\mathcal{I}$ instead of the strategies that are manually designed, i.e., $\boldsymbol{\Omega}$ = {$\omega_{1},\omega_{2}, \omega_{3},\omega_{u}$}, $\mathcal{I}$={$\omega_{1},\omega_{2}, \omega_{u}$}, where $\omega_{1},\omega_{2}, \omega_{3}$ are existing strategies, the network structure of $\omega_{u}$ is existing. In our experiments, $\omega_{1},\omega_{2}, \omega_{3}$ indicate RRT*, BIT*, LazySP.

Thanks for the generally positive comments and constructive suggestions. We have updated manuscript accordingly to make it clearer. We address the corresponding questions below.

1. Pseudocode of the algorithm

We have provided the pseudocodes of all our algorithms in Appendix E of the updated manuscript.

2. Doubts about Table 1: comparison with more challenging benchmarks

Following the reviewers’ suggestion, we generated two groups of new test problems in the 2D environment, i.e., Dense2 and Large2. The environment settings are as follows: (i) Hard2: size, $15\times15$, obstacle density, no less than 0.46. (ii) Dense2: size, $15\times15$, obstacle density, no less than 0.60. (iii) Large2: size, $50\times50$, obstacle density, no less than 0.50. Maintaining consistent parameters across the board, we tested for all methods in new problems. The results are as follows and the complete table have been updated in the revised Appendix D.

Compared to Hard2, operating in the Dense2 environment with a high obstacle density demands increased costs, collision checks, and runtime to reach the goal state. Our method performs best in terms of success rate and collision checks. While RRT* and informed-RRT* demonstrate a low path cost, their success rate lags behind other methods, possibly attributed to their efficacy in solving easy problems. NEXT exhibits lower path costs compared to other learning-based methods, but the collision checks are nearly twice as high.

For large environments, all algorithms demonstrate a notable reduction in success rate. Only a few instances where the distance between the start state and the goal state is not long succeed in finding a path. The majority of failures may be attributed to the limitation of the maximum number of samples, set as 1000. Particularly noteworthy is the superior performance of informed-RRT*, because informed sampling limits the sampling area and reduces the need for samples. For future work, we will try to set a higher maximum number of samples and introduce informed sampling to enhance the performance of our algorithm in large environments.

3. Total running time

We demonstrated the total running time for 1000 test problems across all algorithms to find a successful path. A snapshot of the results is as follows. The complete table is updated in Appendix A of the revised manuscript. To be more clearly represented in the figure, the results are normalized based on the running time of RRT*. HLCA, GNN+smoother, GNN and NEXT are learning-based algorithms.

A common concern about learning-based algorithms is that their time cost due to the frequent calling of a large neural network model at the inference phase, such as NEXT. The running time is higher than non-learning algorithms because our algorithm is improved based on NEXT and runs multiple candidate strategies concurrently. Nevertheless, our algorithm reduces the running time compared with NEXT. Future work could focus on using pre-computation or caching, e.g., reusing past results to overcome some of this overhead.

4. Ablation study about the maximum number of samples

Following the reviewers’ suggestion, we varied the maximum sampling number $N_T$ among {50, 100, 200, 500, 750, 1000, 2000, 4000, 6000} on HLCA. The results are shown in Fig.9 of the updated Appendix B. From the observation of the results, HLCA reaches a plateau in all environments when $N_T$ is 500 in success rate and path cost. For failure cases, robots must generate more samples before successfully finding a collision-free path. This implies an increased demand for collision checks and the computation cost increases on both CPU and GPU.

5. The choice of the thresholds $\alpha$ and $\beta$

Theoretically, larger thresholds $\alpha$ and $\beta$ lead to better results. However, setting a higher threshold will result in longer experiment runtime, with marginal or possibly no performance improvement. During the experimental process, we try to strike a balance between efficiency and performance. $\alpha$ and $\beta$ are set as 0.7 and 0.9, respectively. Besides, we provide details about the hyperparameters of the algorithm in Appendix C of the updated manuscript.

6. Revision of "optimal"

We have revised each “optimal”in the updated manuscript.

7. Concerns related to Neuroscience

Truly, the brain mechanism is the kind of science that can undergo significant revisions in the future. Within the last two years, there is still work in neuroscience that supports the brain mechanism of human decision-making reasoning [1, 2]. Following the reviewers’ suggestion, we make the relevant statement in S2 of the updated manuscript. "The brain mechanism about human decision-making reasoning upon which we draw in this study is confined to the outcomes that preceded our investigation."

We have clarified statements and references in S2. "The current experimental evidence from neuroscience using brain scans suggests that the concurrent reasoning mechanism in PFC includes two concurrent inference tracks [3]."

Following the reviewers’ suggestion, we changed it to "...is designed by imitating the inference buffer of the concurrent reasoning model in PFC [3]." in S3. We changed it to "Motivated by concurrent inference tracks in the concurrent reasoning model [3]" in S4.

[1] Miriam C Klein-Fl ̈ugge, Alessandro Bongioanni, and Matthew FS Rushworth. Medial and orbital frontal cortex in decision-making and flexible behavior. Neuron, 2022.

[2] Anna Cremer, Felix Kalbe, Jana Christina M ̈uller, Klaus Wiedemann, and Lars Schwabe. Disentangling the roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human decision-making. Neuropsychopharmacology, 8(7):1078–1086, 2023.

[3] Ma ̈el Donoso, Anne GE Collins, and Etienne Koechlin. Foundations of human reasoning in the prefrontal cortex. Science, 344(6191):1481–1486, 2014.

8. Presentation about S3.2 and Fig.11, Fig.12

To avoid repetition, only the relevant notations are introduced in S3.2, while the essential components of harmonized self-improving learning are introduced in S4.3. In the updated manuscript, the title of S3.2 has been modified to "Notations". We added the path cost and collision checks in the captions of Fig.11 and Fig.12.

9. Concerns about training set size

Thanks and we have replaced 200 with the parameter $m$ in the updated manuscript and listed $m$ in Appendix C. The training size is also set to 200 in similar work [4, 5]. Chen et.al. [5] conducted further experiments to analyze the effect of the training size. There were 2000 training samples in their experiments. They find with 10% (200) problems of the original training set, the model is robust and obtains the best performance. Therefore, with the same number of training samples, we also set the training size to 200 in our experiments.

[4] Binghong Chen, Bo Dai, Qinjie Lin, Guo Ye, Han Liu, and Le Song. Learning to plan in high dimensions via neural exploration-exploitation trees. International Conference on Learning Representations, 2020.

[5] Chenning Yu and Sicun Gao. Reducing collision checking for sampling-based motion planning using graph neural networks. Advances in Neural Information Processing Systems, 34:4274–4289, 2021.

10. Doubts about Fig.6(c)

Sorry for the inappropriate expression. We've removed this sentence from the updated manuscript. Despite the increase in path cost, we think that the CAM helps improve planning efficiency. Checking collisions with obstacles is vital for motion planning. Hence, we prioritized the more important collision checks in our experiments. Even though the path cost is higher than that of the NEXT in some environments, there is a significant improvement in the collision checks and success rate.

11. Presentation about Appendix A and B

Following the reviewers’ suggestion, we have merged the former Appendix A and B into Appendix A in the updated manuscript. The paragraphs in the former Appendix B have been relocated beside the corresponding table.

Thanks for your careful review and important suggestions for the comparison with a learning-based approache and more challenging environments. We have revised manuscript accordingly to make it clearer. We address the specific questions as follows.

1. Parameters

We have provided details about the parameters of the algorithm in Appendix C of the revised manuscript. These parameters include learning rates, epochs, reliability thresholds $\alpha$, $\beta$, and so on.

2. Total running time

We demonstrated the total running time for 1000 test problems across all algorithms to find a successful path. A snapshot of the results is as follows. The complete table is updated in Appendix A of the revised manuscript. To be more clearly represented in the figure, the results are normalized based on the running time of RRT*. HLCA, GNN+smoother, GNN, and NEXT are learning-based algorithms, others are non-learning algorithms.

A common concern about learning-based algorithms is that their time cost due to the frequent calling of a large neural network model at the inference phase, such as NEXT. The running time is higher than non-learning algorithms because our algorithm is improved based on NEXT and runs multiple candidate strategies concurrently. Nevertheless, our algorithm reduces the running time compared with NEXT. Future work could focus on using pre-computation or caching, e.g., reusing past results to overcome some of this overhead.

3. Comparison with more challenging environments

Following the reviewers’ suggestion, we generated two groups of new test problems in the 2D environment, i.e., Dense2 and Large2. The environment settings are as follows: (i) Hard2: size, $15\times 15$, obstacle density, no less than 0.46. (ii) Dense2: size, $15\times 15$, obstacle density, no less than 0.60. (iii) Large2: size, $50\times 50$, obstacle density, no less than 0.50. Maintaining consistent parameters across the board, we tested for all methods in new problems. The results are as follows and the complete table have been updated in the revised Appendix D.

Compared to Hard2, operating in the Dense2 environment with a high obstacle density demands increased costs, collision checks, and runtime to reach the goal state. Our method performs best in terms of success rate and collision checks. While RRT* and informed-RRT* demonstrate a low path cost, their success rate lags behind other methods, possibly attributed to their efficacy in solving easy problems. NEXT exhibits lower path costs compared to other learning-based methods, but the collision checks are nearly twice as high.

For large environments, all algorithms demonstrate a notable reduction in success rate. Only a few instances where the distance between the start state and the goal state is not long succeed in finding a path. The majority of failures may be attributed to the limitation of the maximum number of samples, set as 1000. Particularly noteworthy is the superior performance of informed-RRT*, because informed sampling limits the sampling area and reduces the need for samples. For future work, we will try to set a higher maximum number of samples and introduce informed sampling to enhance the performance of our algorithm in large environments.

4. Comparison with GNN-TE

Following the reviewers’ suggestion, we have implemented GNN-TE[1] as another baseline method to compare with. A snapshot of the results is as follows, and we will update the manuscript with the complete tables.

Average collision checks.

Average path cost.

After a careful reading, it is noted that the GNN-TE is particularly designed for dynamic environments. With full knowledge of the obstacles, GNN-TE leverages graph neural networks and temporal encoding framework to achieve motion planning problems in dynamic environments. However, our work mainly concentrates on static environments, in which the GNN-TE obtains approximate performance of the GNN. It is expected that GNN-TE will perform better in dynamic environments, which would be our future study topic.

[1] Zhang, Ruipeng, et al. "Learning-based Motion Planning in Dynamic Environments Using GNNs and Temporal Encoding." Advances in Neural Information Processing Systems 35 (2022): 30003-30015.

5. Doubts about UR5

There are two reasons why this happens. Unlike other robotic arm environments, obstacles are mainly concentrated around the robotic arm in UR5. On the one hand, RRT* samples a state from free space, and determines a new state to connect to through the steer function. The steer function only generates states that are close to the search tree within neighboring areas. Therefore, in environments with high obstacle density, tree-based sampling methods such as RRT* struggle to expand new states. On the other hand, the maximum sample number is limited to 1000 in our experiments. Under this condition, RRT* reaches the maximum budget of the sampling nodes and is forced to be terminated without finding a solution. Consequently, it directly results in a substantial decrease in the success rate of RRT*.

6. Minor comments

We have improved the typo mentioned in minor comments in the revised manuscript.