Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity

We feel gratitude for the valuable insights and suggestions, and below is our detailed response.

---

Q1: More complex imitation tasks can enhance the expressiveness of the method.

A1: We have added experiments with additional tasks and measure. Please refer to our response to Q3 of reviewer Cxuk.

---

Q2: It would be helpful to add sensitivity studies on hyperparameters.

A2: We have conducted a hyperparameter study for the $q$ hyperparameter, the scale of the EBC reward; these results are mentioned in Appendix E.4. Other hyperparameters follow the settings in original implementations since these are not newly introduced by our method.

---

Q3: Source code not provided.

A3: We will open-source the code and seeds when the paper is published.

---

Q4: Important concepts should be briefly introduced, e.g. MAP-Elites, VPPO, etc.

A4: The details of MAP-Elites can be found in Section 2.1. For VPPO, we only briefly mention it in Section 2.1 as Vectorized PPO in the context of PPGA and in Appendix A2 l.671-673. We will expand the explanation both in the main text and in the appendix.

---

Q5: The measure function m defined in the experiments does not seem to reflect curiosity-driven exploration.

A5: The measure value itself is not the curiosity. Instead, whether or not the diverse measure has been visited represents the behavioral-level curiosity.

---

Q6: Imitation tasks based on real-world data (e.g., human motion capture data) could better highlight the advantages of the proposed method.

A6. We have conducted additional experiments on various imitation learning locomotion tasks (see our response to Q3 of reviewer Cxuk), and we are happy to extend our method to imitation tasks based on real-world data in our future work.

---

Q7: How is a sufficiently explored region defined?

A7: The authors see that the “sufficiently explored region” is mentioned on l.240 page 5. We will remove the “sufficiently” from the text. Basically, as soon as the region is visited, the EBC reward will be zero for consequent visits, to facilitate exploitation of high-performing policy in this region.

---

Q8: From the perspective of imitation learning theory, how can the learned policy outperform the expert policy?

A8: When we talk about performance we typically talk about the QD score. Hypothetically, it is possible that the expert policy has a higher cumulative reward since the expert would not be the optimal policy. However, we do not claim that the learned policy outperforms the expert. In Humanoid (see Fig.2), we do observe a higher QD score and coverage compared to PPGA with true reward. However, as shown in Fig.4, none of the techniques can outperform PPGA with true reward with EBC.

---

Q9: Is there an optimal balance between imitation rewards and EBC rewards?

A9: We studied the scale of the EBC reward, $q$ in Appendix E.4. $q=2$ appears to be the best choice. A sufficiently high $q$ is needed to for the behavioral level exploration to encourage higher QD score. It is impossible to find the exact optimal choice since the q value is continuous and there are infinite choices, and the optimal choice varies from tasks to tasks, so we choose q=2 in our experiments.

---

Q10: What happens if the expert demonstrations are of low quality?

A10: Because the demonstrations are selected to be diverse, the expert data are already sub-optimal. If the performance is much lower than even a local optimum solution, then it seems difficult to achieve policy improvement. However, this can then be said for all imitation learning methods since it is assumed that the demonstrations are of desirable quality (otherwise there is limited point to imitating them).

---

Q11: The improvement in the Halfcheetah and Walker2d tasks does not seem significant.

A11. We have performed a Tukey HSD Test on Table 7; comparing our EBC-enhanced algorithm to their counterparts, and we find that 7/9 effects are positive and 5 of these are significant with $p \leq 0.05$. Also refer to our answer to Q4 of U4H2 for further explanations.

---

Q12: Mutual-information-based methods also achieve diverse behavioral policies. What are the advantages of QD-IRL?

A12: The mentioned references assume it is possible to sample skills from a skill distribution, and then condition policies based on the given skill variables. In our work, the measures are not known in advance and are determined from the evaluation. The key difference is that we can optimise diversity across a particular measure rather than be limited to a particular skill distribution. So typically, unless the distribution of said techniques is uniform across the entire feasible space, QD algorithms will have more emphasis on diversity. We will discuss this difference in the updated paper.

---

We hope our response fully address your concerns.

We appreciate the valuable insights and suggestions, and below is our detailed response to address your concerns.

---

Q1: More diverse tasks other than locomotion will be beneficial.

A1: While we limit the scope of the experiments to robot locomotion tasks, we do agree with the inclusion of much more diverse tasks to demonstrate the capability of our algorithm. In particular, we have extended our method to more diverse tasks and will add these contents in the revised paper. It is the same table also mentioned to reviewer Cxuk in Q3 (reproduced here for convenience):

---

Q2: The authors never refer to how they construct the archive for the QD algorithm.

A2: We use the archive construction rules based on MAP-Elites, which is mentioned in section 2.1.

---

Q3: It would be helpful to introduce the behavior descriptor for each experiment, and the details of each tasks.

A3 : The information about the behavior descriptor is referenced at various locations in the paper under the terminology “measure” (which is the terminology that was used in the PPGA paper). Specifically, the measure used is highlighted in Section 4 as follows: “the measure function maps the policy into a vector where each dimension indicates the proportion of time a leg touches the ground.” Moreover, we have clarified definition of state (policy’s inputs) and action (policy’s outputs) of each task in our revised paper.

---

Q4: The general improvements in performance EBC has to offer, in some cases are very minimal.

A4: Actually, when comparing performance, we should compare the base method with this method plus EBC (e.g., GAIL vs. GAIL-EBC) in terms of the QD-score. In the QD-score plots, the authors can only observe 2 comparisons among 9 comparisons (3 pairs $\times$ 3 environments), namely the Halfcheetah for DiffAIL-EBC vs DiffAIL and VAIL-EBC vs VAIL, that are not significantly better. We highlight that EBC improves the QD score of GAIL, VAIL, and DiffAIL by up to 185%, 42%, and 150%, and that the standard deviations are very small. We have performed a Tukey HSD Test on Table 7; comparing our EBC-enhanced algorithm to their counterparts, and we find that 7/9 effects are positive and 5 of these are significant with $p \leq 0.05$.

---

Q5: It would be helpful to provide videos to illustrate the learned diverse solutions.

A5: Thanks for the suggestion. One of the benefits of diversity is that the archives can be used for adaptation to new environment. In the updated submission, we will include a video of adaptation to new environments which can demonstrate the types of behaviors in the evolved archives as well as how this diversity can be exploited.

---

Q6: Is the EBC values bounded?

A6: Since the unscaled EBC reward is either 0 or 1, so the scaled EBC rewards are bounded in $[0,q]$, where $q$ is the hyperparameter that controls the weight.

---

Q7: Could the authors explain how they did the visualization for Fig. 3 and elaborate a bit more on what this figure represents?

A7: Fig.3 represents the overall QD performance of the archive using a heatmap. The colors represent the fitness levels (quality), while the spread across the two-dimensional measure/behavior space represents the diversity.

---

We sincerely hope these responses fully address your concerns.

Thank you for the valuable suggestions. Here is our response to questions and concerns.

---

Q1. This paper lacks evidence support for the significance of QD (e.g. the ability of adaptation).

A1. The claim that QD approaches help with adaptation in changing environments is reasonable and supported by prior works. We do notice that on l.46, the given reference does not adequately reflect the adaptation scenario, so we provide a more suitable few references. While the claim is not central to our work, we perform few-shot adaptation tasks for Humanoid-hurdles, and find that GAIL-EBC significantly outperforms GAIL on the return, and is comparable to the original QDAC performance of Grillotti et al. :

---

Q2. More diverse IL tasks are needed to demonstrate the generalizability of the method.

A2. We have conducted additional experiments with diverse measures. Please refer to our response to Q3 of reviewer Cxuk for details.

---

Q3. While GAIL and VAIL rewards are updated at every iteration, the finesses stored in the archive are not regularly updated (as acknowledged in the limitations section, lines 907-910).

A3. That's indeed the limitation of our work that we already acknowledged, and we are happy to address this issue in our future work. Moreover, Algorithm 2, which dynamically adjust the fitness scores, mitigates the impact of this problem by computing a running average of past elite fitnesses.

---

Q4. EBC reward is non-Markovian and depends on whole trajectory, which may affect the theoretical guarantee.

A4. The combination of non-markovian reward and PPO are explored and backed in prior work. For example, [1] validated that episode-based reward structures, when properly decomposed into each step, enhance PPO’s convergence and sample efficiency. Hence, we safely assume the convergence guarantee of PPO in our Lemma 3.1. Empirically, we also validated the effectiveness of EBC reward.

[1] Arjona-Medina, Jose A., et al. "Rudder: Return decomposition for delayed rewards." Advances in Neural Information Processing Systems 32 (2019).

---

Q5. BestReward of EBC variants takes longer to converge (which is normal, as PPGA now optimizes for fitness+q×EBC).

A5. In general, the higher the coverage, the longer it will take to evolve the elite solutions for all the covered cells. We will mention this in the text on page 7, l.364, in the text about Fig.2.

---

Q6. The computational requirements seem overly demanding (48 hours on 4 A40 GPUs per experiment).

A6. We apologize that the text is not clearly written. We use 1 GPU per experiment but run multiple experiments at the same time. We will rewrite this part.

---

Q7. This method should not be named as a framework. As presented, the EBC reward seems limited to the Gradient Arborescence subfamily of QD approaches (like CMA-MEGA and PPGA) and may not be applicable to other QD algorithms.

A7. This is a limitation that we have acknowledged in Appendix G. Please note that we have implemented the EBC reward on different imitation learning methods, showing why it is usefully conceived as a framework. While MAP-Elites does not have an RL problem, PGA-ME and DCRL seem compatible for our framework. For instance, the EBC reward would allow to take a step in the direction of the policy which generates new measures. Given the wide range of experiments, we leave this to future work.

---

Q8. The connection of CMA-ME and EBC regarding the exploration should be discussed.

A8. We claim that EBC allows a policy gradient algorithm to update the policy towards empty behavioral areas directly, which is more effective compared with the exploration of CMA-ME which doesn't utilize the gradient information. For more details, please refer to Q2 from Reviewer Cxuk.

---

Q9. Figure 4 is redundant with Figure 2 - the additional variant could simply be included in Figure 2.

A9. Figure 4 is for more straightforward comparison to show specifically that EBC can improve PPGA with true reward on conditions without EBC. We separated these so it is more easy to discuss in dedicated paragraphs in the text.

---

Q10. Policy archives in Figure 3 have inconsistent colorbars.

A10. Please note that Figure 3 is plotted only for comparing the coverage metric, not the fitness (l.392). Therefore, the color (which represents fitness) is not our focus.

---

Q11. Consider reporting CCDF plots to better characterize the distribution of episode returns in the final policy archives.

A11: Thanks for suggestion. We will report the CCDF plot in our revised paper.

---

We sincerely hope these responses fully address your concerns.

Thanks for the valuable insights and suggestions.

---

Q1. It would be better to discuss the relationship between Yu et al.'s paper.

A1. The key difference between Yu’s work and our work lies in two aspects:

• Yu’s work adopts a ​single-step reward bonus (calculated per (s,a) pair), while our method uses ​episode-based reward bonus aligned with whole-episode measurements.
• Yu’s work focuses on QD-IL, whereas our framework extends to QD-RL.

We will add these discussions in our revised paper.

---

Q2. Why use EBC given CMA-ES already handles quality/diversity?

A2. The EBC reward can supplement the PPGA algorithm: while PPGA works by evolving random coefficients for each measure dimension, the EBC reward encourages the policy to unlock new areas of measure space by adding it to the fitness, which is independent of the local measure space, for a more direct, global, and efficient search of new behavioral areas.

---

Q3. More experiment needed on tasks with different measure dimensions and type of measures.

A3. We tested EBC across tasks with varying measure dimensions (m):

• Jump (1D: lowest foot height)
• Angle (2D: body angle)
• Feet-contact (4D: leg-ground time)

GAIL-EBC consistently outperforms GAIL in all scenarios:

---

Q4. How do demonstration(demo) quantity/diversity/quality impact performance?

A4. Experiments with varying demo counts in Humanoid environment:

Key observations:

• With 1 demo, GAIL-EBC’s QD-Score drops by 15% vs. 10-demos.
• Using non-diverse elites (top4 best-reward):

---

Q5. It would be helpful to add the pure IRL algorithms (without PPGA) with and without EBC bonus as baselines.

A5. Without the QD algorithm, there would be no QD-Score, coverage, and average reward metrics. So we would only be able to compare the maximal fitness, which is not the our objective.

---

Q6. It would be helpful to use statistical tests to show the significance of the results.

A6. We perform a Tukey HSD Test on Table 7; comparing our EBC-enhanced algorithm to their counterparts, 7/9 effects are positive and 5 of these are significant with $p \leq 0.05$.

---

Q7. The code is not available.

A7. We will open-source the code and the seeds when the paper is published.

---

Q8. PPGA also optimizes behavior diversity and aims to maximize the QD-Score. Why does PPGA with EBC achieve a better QD-Score than PPGA without EBC?

A8. We kindly refer you to A2. PPGA itself is not sufficient in exploration. However, our EBC bonus synergized with CMA-ES could facilitate exploration and exploitation more effectively (see our paper line 244).

---

Q9. Why does DiffAIL-EBC outperform PPGA with true reward in Humanoid?

A9. As shown in Fig.4, PPGA itself's performance improves significantly with EBC (improved exploration). However, DiffAIL-EBC does ​not surpass PPGA-EBC, aligning with expectations. See analysis in l.420-428 (Page 8).

---

Q10. Is comparing average rewards necessary?

A10. While QD-score is the primary metric, average reward provides insights into archive quality. We retain it for supplementary analysis but emphasize QD metrics as the core evaluation.

---

We hope these answers address your concerns.