Quality Diversity Imitation Learning

In the QD community, robotic locomotion tasks in MuJoCo are commonly used, such as PPGA[1], QDAD[2]. While Atari games are common in RL and IL, their implementations do not transfer to brax, which is the framework we use for hardware-accelerated simulation experiments.

[1] Batra, S., Tjanaka, B., Fontaine, M. C., Petrenko, A., Nikolaidis, S., & Sukhatme, G. (2023). Proximal policy gradient arborescence for quality diversity reinforcement learning. arXiv preprint arXiv:2305.13795.

[2] Grillotti, L., Faldor, M., León, B. G., & Cully, A. (2024). Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics. arXiv preprint arXiv:2403.09930.

---

Q5: Could the author give more analysis on the learned behavior patterns? Do they really correspond to "skills" in the semantic level? E.g. moving left leg, moving right leg, etc. It should be more convincing if these diversified policies can be visualized and validated as different behavior patterns for better interpretability.

A5: This is indeed a crucial point! Actually, behavior patterns can be regarded as skills or as behavioral variations of a skill, depending on one’s definition. To exemplify how a useful skill can be evolved in our archive, we note that in the Humanoid environment, our algorithm learns two distinct behavior patterns with one only using the left leg and another only using the right leg to move forward. These two can be regarded as distinct skills, particularly in terms of dealing with changes in the environment: if some issue caused the left leg to be broken, then the learned “skill” enables the robot to move forward only using the right leg.

To visualize the behaviors, we point to the videos in Supplementary Material, which demonstrate 3 different learned behaviors in each environment, chosen based on their distance to each other. For example, the Humanoid task behaviors include one moving in very small forward steps (behavior 1), one moving sideways (behavior 2), one leg shuffling (behavior 3), and one jumping behavior (behavior 4).

---

We hope that our responses can fully address your concerns and will be grateful for any feedback.

Thank you for your time and valuable suggestions. Here are our detailed responses.

---

Q1: The reviewer mentions various references regarding Multi-domain Imitation Learning and skill- or option-discovery [4] [5] [6] and seeks to clarify how these settings are different.

A1: Thanks for providing these work. In multi-domain imitation learning [1,2,3], the aim is to form a context-conditioned policy that performs well across different tasks. Our setting is different as we are interest in designing a large archive of policies that represent diverse behaviors.

In option discovery and related methods [3,4,5,6], the aim is to form low-level policies to be used within a higher-level MDP (e.g. an MDP with options). This line of work is related to hierarchical RL, where a low-level policy can be initiated under certain states and terminated upon other states (like a subgoal). In our approach, we do not seek to form policies based on their utility in a hierarchical setup; instead, we seek to form a maximally diverse and high-performing set of policies.

---

Q2: Currently, the learned diverse policies can only be read by the curves (like that in Fig.1.(a)). There lacks a formal and precise theoretical interpretation of how and why it is connected to diverse behavior patterns. How strongly does a local optimal suggest a behavior pattern and how well it is guaranteed? It would be helpful to develop more theoretical analysis in this regard.

A2: Sorry for the confusion. Actually, Fig. 1 is merely an illustration of our motivations. Each policy corresponds to a particular behavior, and Fig. 1(a) illustrates a problem with the traditional approach of PPGA. While PPGA stores high-performing policies for each behavioral region it explores, it overemphasizes particular regions of the behavior space. Introducing the measure bonus helps to improve this exploration process. Our approach naturally leads to high-performing solutions for diverse behaviors, thereby optimizing the objective in Definition 1. A theoretical motivation for the measure bonus can be found in Lemma 1. The main purpose is to encourage visiting new cells in the archive, thereby increasing the coverage of the behavior space.

As for the learned policy archives, these can be found in Figure 6, where for each behavior in the archive we display the performance. Moreover, a directly applicable, practical evaluation method for Definition 1 is the QD Score, which we show our QD-IL approach is consistently optimizing.

To improve the clarity of Fig. 1(a), we have improved the caption to illustrate the motivation that PPGA does not sufficiently encourage exploration in itself, and why the measure bonus can help.

---

Q3: How sensitive is QD-IL to the choice of hyperparameters? It would be helpful to have a robustness/sensitivity test.

A3: Thanks for the suggestion. As our hyperparameter study shows (see Fig.10 in the updated paper), our results can be further improved by tuning the p and q hyperparameters related to our measure bonus. For your convenience, we hereby show the result table:

We found that p=0.5 q=1 is the most promising hyperparameter.

---

Q4: Can the framework be adapted to domains with high-dimensional input data, other than the three environments stated in the paper? It is also questionable to say that the "most challenging" humanoid environment, since a pixel-based input environment like Atari is clearly more challenging. It would be helpful to make it clearer.

A4: Thank you for the valuable concern. We agree that there are more challenging environments. Actually, this statement was there to highlight it was most challenging within the widely-used Mujoco environments. Since this is unclear in the abstract, the mention of most challenging has been removed.

Q1. I am still concerned with what is exactly the benefit of only learning a diverse of policies, without considering how to leverage them in the most efficient way? Moreover, learning a context-based policy does not necessarily be in a multi-domain setting, it can also be applied to the current setting where the agent is supposed to finish one single task and show diverse behavior patterns.

A1. In terms of what archives of diverse policies can be useful for, a key application of diverse policies obtained from QD algorithm is adaptation to new environments or new robot configurations. For instance, the sensors or actuators of a robot may be altered in some way, in which case performing optimisation over the archive can yield high-performing policies more quickly. This line of work has been widely studied (see e.g. [2] for one of the original studies). To give an example with Humanoid, if one leg is damaged, jumping behavior with the other leg will help the Humanoid agent to move forward.

With the adaptation application above in mind, context-conditioned policies may be useful in settings where one wants to use the archive for adaptation and one can exploit a relation between the context and the optimal behavior measure. Applying the MConbo-IRL framework in such settings will be interesting for future work, which we mention in the future work discussion in the pdf.

Q2. In the QD-IL setting, I noticed a recent paper that hasn’t been discussed in the literature. Could the authers include the discussion for this? [1]

A2. Subsequent to our work, this paper is a recent algorithm for QD-IL, which follows our setting and was submitted to arxiv one month after our ICLR submission. While the paper follows our work closely, in terms of recognising the significance of encouraging behavior-space exploration and taking into account the measure in the design of the reward function, there are a few key differences. In particular, they introduced a measure-based bonus for single-step archive exploration, while we compute the episode-level measure bonus from the archive more straightforwardly. Moreover, they stabilized the training of adversarial imitation learning by applying Wasserstein auto-encoder (WAE) with latent adversarial training. The measure conditioning is used for learning latent variables through the reconstruction in WAE, which is different to our method that applied measure conditioning for discriminator training. We have included the discussion of [1] in our updated paper.

[1] Yu, X., Wan, Z., Bossens, D. M., Lyu, Y., Guo, Q., & Tsang, I. W. (2024). Imitation from Diverse Behaviors: Wasserstein Quality Diversity Imitation Learning with Single-Step Archive Exploration. arXiv preprint arXiv:2411.06965.

[2] Cully A., Clune J., Tarapore D. & Mouret J-B. Mouret (2015) Robots that can adapt like animals. Nature, 521, pages 503–507.

Q6: In Figure 1(b) Multiple behavior demos, traditional behavior cloning has the mode averaging problem, but some current methods such as diffusion-based imitation learning has the ability to learn multi-modality, right?

A6: In Fig.1b, we pointed to the single-policy as experiencing a kind of mode averaging behavior. The reasoning is that a single policy can only represent a single point in the policy/behavior space. Therefore, designing the policy in a different manner, e.g. using a diffusion model, would not counter this issue. We kindly request that, could you please clarify the question and/or suggest some references?

---

Q7: QD-IL relies on the definition of behavior diverse measures m. In Chapter 4.2.1, it says that the measure abstracts high-level task features. Does this mean you always need to design task-specific behavior patterns in advance? I would assume this is very hard for more complex and long-term tasks.

A7: Thank you for this valuable feedback! As we mention in A1, the behavior should be defined in advanced but it can be done either manually, by using various statistics based on the needs of the user, or using automated techniques, where for instance an auto-encoder could be useful. In most cases though, in quality-diversity optimization, the behavior measure is designed by hand, i.e. by choosing dimensions which the user is interested in from a diversity perspective. In the MuJoCo tasks, the chosen features are based on earlier works (e.g. PPGA), which uses for each leg the proportion of the time it touches the ground. Such a design leads to useful diverse policies, e.g. one may obtain a policy that only uses the left leg, which is useful in case the right leg is broken.

In long-term tasks, one may use some statistical properties of the task. For Markovian Proxy Measures, the present measure we use, i.e. the proportion of touching the ground, it does not depend on the duration of the task. For long-term episodic measures, for instance, one may compute how frequently particular states have been visited, the spread of states across the episode, etc.

---

Q8: In Figure 4, what is the x and y axis?

A8: The x axis is the proportion of time Leg 1 touches the ground and the y axis is the proportion of time Leg 2 touches the ground. Leg 1 is the left leg in Humanoid and Walker2D and it is the front leg in Halfcheetah. Leg 2 is the right leg in Humanoid and Walker2D and it is the back leg in Halfcheetah. This description has been added to the caption of Figure 4 for improved clarity. Thanks for your concern.

---

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

Thank you for your time and valuable suggestions. Here are our detailed responses.

---

Q1: The measure bonus calculation requires high-level task features design, which could be challenging for long-term and complex tasks. I would assume it always needs task-specific design.

A1: This point is critical. Actually, we need to clarify that the key task-specific design of complex tasks is to define the mapping from state to measure. After this mapping is well-defined, the calculation of measure bonus will be simple. For the mapping,

• Typically, this is done based on manual design in terms of what the user is interested in. For example, in Mujoco environment, each element of single-step measure is simply designed as whether the corresponding leg touches ground (please think about it: this is actually a mapping from state to measure). This design well suits our needs, because we naturally hope the robot to constantly move forward without falling. In case of one leg broken, the robot can switch to another leg to move on.

• Due to it being a mapping from state to measure, it is also possible to design its features automatically, e.g. using auto-encoders to obtain a low-dimensional representation, such as the technique in [1].

Hence, the task-specific design is not problematic.

[1] L. Grillotti & A, Cully (2022). Unsupervised Behaviour Discovery with Quality-Diversity Optimisation. IEEE Transactions on Evolutionary Computation.

---

Q2: I can’t find the details about how the authors define the behavior spaces for the 3 tasks. Maybe I missed something, but I would appreciate it if the authors could explain more about how to implement the behavior measures in the experiments.

A2: Sorry for the confusion! The definition of measure (behavior space) is at the caption of Figure 2 and the description of the experiment setup (Section 5.1). In Mujoco setting, the measure function is a vector where each dimension indicates the proportion of time a leg touches the ground. However, the measure function can vary based on specific needs in real world application. We have improved the description of the behavior space used in the experiments.

As in A2, the single-step behavior measure is implemented by a function mapping from state into a vector with each entry a indicator value (whether the corresponding leg touches ground in this state), and the episodic measure is the average of the measure of each state in this episode.

---

Q3: The experiments do not contain tasks with manipulation. It would be great to show how the QD-IL scales to other robot manipulation tasks.

A3: This is a great extension. However, robotic locomotion tasks in MuJoCo are commonly used in QD community, e.g., in PPGA [1], QDAC [2]. While there are tasks such as air hockey and grasing, these are not available in standard implementations of manipulation tasks in brax, which is the framework we use for hardware-accelerated simulation experiments.

[1] Batra, S., Tjanaka, B., Fontaine, M. C., Petrenko, A., Nikolaidis, S., & Sukhatme, G. (2023). Proximal policy gradient arborescence for quality diversity reinforcement learning. arXiv preprint arXiv:2305.13795.

[2] Grillotti, L., Faldor, M., León, B. G., & Cully, A. (2024). Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics. arXiv preprint arXiv:2403.09930.

---

Q4: Equation 3 lacks the explanation of $\tau^E$

A4: We apologize for this undefined notation. The notation was meant to indicate state-action pairs are sampled from the demonstration set. The $\tau^E$ notation has been changed to $\mathcal{D}$ for consistency with the rest of the document.

---

Q5: The reviewer wanted to know more about the meaning of the policy space in Figure 1(a).

A5: Sorry for confusion. Actually, figure 1(a) illustrates the fitness as a function of the policy space, i.e. the space of possible parameter settings of the policy. The figure represents the limitations of traditional policy gradient for QD optimization, namely that it will tend to explore only regions with high fitness, ignoring a large diversity of policies.

The behavior space is directly correlated to the policy space. Since it is more straightforward to give an example based on the behavior space, we now use the behavior space as the x-axis so that we can easily explain the issue and relate it to the example in Figure 2. In short, traditional policy gradient techniques would only cover particular settings of leg contact ground times, e.g. jumping forward and walking fast, as these are highly fit, but not include more diverse behaviors that may be of interest.

---

Q1. From my understanding, QD-IL is then mainly designed for robotic locomotion tasks. If it can only be used in this setting, it is not very convincing.

A1. Robotic locomotion is a common benchmark in quality diversity (QD) optimization and quality diversity reinforcement learning (QD-RL), since there are well-implemented simulation environments to support QD learning. Our QD-IL framework is a generic method for learning diverse and high-performing policies from limited demonstrations, which is not by any means limited to robotic locomotion. The authors stress that the issue we mentioned is one of software compatibility rather than of algorithmic applicability. We are happy to apply our QD-IL framework to robotic manipulation tasks once such benchmarks are implemented in brax. At present, the tasks of air hockey and grasping, as mentioned in one of our previous responses, are implemented in other software that is not compatible with brax.

Q2. Regarding Q6, I thought the multi-modality refers to the D3IL[1] and DiffusionPolicy[2] definition, which is the different solutions given a task or multi-modal action distribution.

A2. Thank you for suggesting these two papers. We can see that techniques such as diffusion policies and behavioral cloning have varying degrees of behavioral entropy according to the benchmark evaluation of D3IL, which is a dataset with demonstrations that are uniform across the behavior space.

We note that the behavior entropy does not necessarily imply a set of behaviorally diverse policies, even though behavior entropy may help to search for diverse policies. In other words, high behavioral entropy policies are highly random in the behavior space, which implies different trials may lead to different trajectories each with different behavioral measures. This is not necessarily a desirable property as an end-goal; for instance, if each trial a different trajectory is obtained, then the policy is likely to be dangerous or unpredictable. However, we may see some role for high-entropy policies in terms of exploring the behavior space, for instance, using behavioral entropy as a bonus (very related to our measure bonus). Note that we actually considered using such a bonus but did not find a suitable implementation for our purposes.

[1] Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations, ICLR’24

[2] Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS’23

Q5: The reviewer mentions that using only 4 demonstrations may lead to high variance in the results and that clarifying the reliability of the results would be helpful.

A5: That point is really valuable! Actually, in our original submission, we had already provided the study of number of demonstrations. Please refer to Figure 12,13 in Appendix H, where we set the demonstration number to 4,2 and 1. In addition to these experiments, we have done further experiments where we increase the number of demos to 10, as in Figure 14 in updated paper. The result shows that the performance significantly drops when the number of demo decreases from 2 to 1, and the performance significantly improves when we set the number of demos to 10 (suggesting lower variance with more demonstration). However, even if we only have 2 but diverse demonstrations, our method can outperform PPGA with true reward, suggesting our method's ability to learn from diverse but very limited demonstrations.

---

Q6:. More discussion of the new approach needed for raising score.

A6: Thanks for your suggestion. In addition to the above-mentioned experiments, we also point to further experiments which show that the measure bonus can in principle be applied regardless of the form of reward function (whether it is true reward or learned reward), which means the measure bonus can not only improve QD-IL but also improve QD-RL. We experimentally sought to verify this on the Humanoid problem (please see PPGA-trueReward-Bonus in Fig.16 in the updated paper).

For your convenience, we provided the summary of results here:

where ''ExpertBonus'' in IL method means the demonstrations are sampled from the expert policy which is learned by PPGA with measure bonus. The results confirm that the measure bonus significantly improve PPGA with true reward. Based on that, we conclude that the measure bonus can both significantly improve PPGA for QD-RL problems and improve QD-IL method.

The rationales of improving these two are similar: the measure bonus helps to mitigate the local optima issue. Please note that the local optima is not such problematic when we only seek to find only one optimal policy (which is the traditional RL setting). However, in QD-RL setting where we aim to find multiple diverse solutions which collectively maximize the fitness, the local optima issue will be very problematic since we need to optimize the decision variable (policy parameter) in each sub-region (as illustrated in Figure (1)) but the policy parameter will intrinsically stay around local optima point due to the nature of gradient-based optimization method.

Furthermore, in the QD-IL setting, this issue is further exacerbated due to the discriminator will shape the reward based on limited demonstration, so the reward will be much more localized. Hence, the measure bonus is designed to mitigate the problem caused by local optima, which is the common issue of QD-RL and QD-IL.

Additionally, please note that the performance of MConbo-GAIL even exceed PPGA with true reward, which is surprising. We interpret this as the local optima issue mentioned before in ''PPGA with true reward'' is not solved, leading to sub-optimal performance. What verified our interpretation is that, the performance of ''PPGA with true reward'' with measure bonus outperforms MConbo-GAIL, which means the IRL can hardly outperform RL with true reward, when both are equipped with measure bonus.

---

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

Thank you for your time and valuable suggestions. Here are our detailed responses.

---

Q1: I think an ablation on the demonstrations used to verify the robustness of the proposed method could be important. It seems that a fixed set of 4 demos per environment tested are used.

A1: That's a good point. Actually, we already performed an analysis of the number of demos in {1,2,4} (please see Appendix H). In addition to this study, we have added an additional 10 demonstration experiment in Humanoid (see Fig.14 in the updated paper), which gives a higher QD score. In contrast, if the demonstration number decreases from 2 to 1, the performance significantly drops (as in Figure 13 (a) in Appendix H). These result aligns with our expectation, since our setting is actually "learning diverse behavior from limited but diverse behaviors" and it is important for the expert demonstration to be as diverse as possible, to facilitate learning diverse behaviors.

In general, we can conclude that the more diverse the demonstration, the higher the QD-score that will be obtained. However, we primarily aims to showcase the capability of our method to learn diverse policies from only a limited numberof demonstrations, so we only use fixed 4 demonstrations in our experiments.

---

Q2: The reviewer is interested in the correlation between the demonstrations and the observed behaviors for the different included algorithms.

A2: Thank you for your valuable concern. To compare PPGA with true reward and our proposed QD-IL algorithm in terms of the behaviors their respective archives contain, we kindly request you to point to Figure 6, where one can observe the diverse behaviors filling the archive with high-performing behaviors. This figure shows the behavior-performance maps, where for different 2-dimensional behavior vectors the respective elite fitness value are shown. Since there are only four demonstrations, nearly all of the policies in the archive are highly distinct from the demonstrations. The archive heatmap is commonly used in QD task literature to compare the learned behaviors (for example, paper [1][2][3]).

In terms of interpreting the behaviors, we point to the videos with diverse behaviors in Supplementary Material.

[1] Batra, S., Tjanaka, B., Fontaine, M. C., Petrenko, A., Nikolaidis, S., & Sukhatme, G. (2023). Proximal policy gradient arborescence for quality diversity reinforcement learning. arXiv preprint arXiv:2305.13795.

[2] Fontaine, M., & Nikolaidis, S. (2023, July). Covariance matrix adaptation map-annealing. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 456-465).

[3] Fontaine, M., & Nikolaidis, S. (2021). Differentiable quality diversity. Advances in Neural Information Processing Systems, 34, 10040-10052.

---

Q3: Are there any training / wall clock time numbers of the proposed method? Given that a gpu sim like brax is used I think wall time may be somewhat relevant to be aware of as the purpose of GPU sim is to improve training wall time.

A3: Thanks for your valuable concern. We have recorded the wall time of each iteration in PPGA and MConbo-GAIL in humanoid environment. PPGA costs 121s and MConbo-GAIL costs 150s for each iteration. Both methods use brax to accelerate the simulation.

---

Q4: How well might this method fair with sub-optimal demonstrations? Or would some form of offline RL approach now be needed?

A4: Sorry for the confusion. Actually, our demonstrations are already sub-optimal. Our procedure to generate demonstrations is as follows. First, we select the solutions in the archive with the top 500 highest scores. Second, from those we enforce a behavioral diversity criterion such that these demonstrations are evenly spaced in the behavior space. This leads to very diverse and therefore sub-optimal solutions. So in general, the expert data are sub-optimal with regard to the global fitness optimum. Some of the solutions will be close to local optimum solutions.

Furthermore, we think offline RL would not be possible since we do not have the reward function.

Q1. Thank you for clarifying however I still have some concerns. Since the performance of the method on just one demonstration is already quite high in appendix H, it would appear this method should be tested on harder benchmarks where 1 demonstration is insufficient.

A1. We selected these environments as these are standard ones where our baseline, PPGA with True Reward, would perform well. As one can observe in Appendix H, Figure 13 shows a 25% improvement for 4 demonstrations vs 1 demonstration. This improvement is many times the standard deviation of the 1 demo data as can be observed from Figure 13.

Q2. Moreover regarding the interest in correlation between demonstrations and observed behaviors, while the archive filling up is good, it's hard to semantically understand as the 2D behavior vectors seem to be not interpretable. The videos are not clear as well as it sort of looks like mostly similar behaviors in one task and there aren't any videos of the demonstrations. Some direct qualitative comparisons in the supplemental or better the main paper would be great between demonstrations and learned behaviors.

A2. The authors have previously mentioned a comment relating to the behaviors for Reviewer P8f4, which we quote here:

To visualize the behaviors, we point to the videos in Supplementary Material, which demonstrate 3 different learned behaviors in each environment, chosen based on their distance to each other. For example, the Humanoid task behaviors include one moving in very small forward steps (behavior 1), one moving sideways (behavior 2), one leg shuffling (behavior 3), and one jumping behavior (behavior 4).

Since we did not yet add this explanation to the pdf, we will do so in the next revision.

Q3. Moreover given how fast brax experiments run (just a few minutes) I hope it is not too difficult to test on the harder manipulation/locomotion tasks in brax.

A3. As one can see in the pdf, in the first paragraph of Section 5.1, it can be read that each experiment takes 2 days (assuming full availability of server). The reviewer may be used to traditional RL or imitation learning; the QD-RL and QD-IL frameworks require training a very large number of policies rather than just a single one, thereby increasing the total training time.

Overall, we hope that our responses can fully address your concerns and will be grateful for any feedback.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give to us.

Q4: The agent will learn a reward from each expert behavior separately and there is no shared information, which put GAIL and VAIL in intrinsic disadvantage. It may be worthwhile to explore additional forms of baseline methods based on standard GAIL/VAIL. For example, applying PPGA + GAIL/VAIL to create a policy archive for each demonstration expert individually, then aggregating all policy archives together, could provide another baseline.

A4: This point is thought-provoking! However, our technique also learn one single reward function, which can be sensitive to the different experts based on the measure conditioning, and we also only use a single archive. Therefore baselines such as PPGA-VAIL/GAIL are at no intrinsic disadvantage.

The suggested baseline would have a few disadvantages compared to our present implementation. First, the technique has limited memory scalability since for each expert one would need a separate archive. Second, separating the optimization process into different, independent optimizers limits the interchange of solutions, and it may be hard to find the genotypic hypervolume that contains the high-performing and diverse phenotypes within the same joint budget of function evaluations.

---

Q5: The reviewer shared some concerns about the writing, including more clearly defining terms, pointing to references, and clarifying aspects of the pseudo-code.

A5: Thank you for the helpful feedback. We agree that these points could be improved. We have added more background on QD algorithms, including canonical algorithms, more details on how MAP-Elites works and how the grid-based archive is organized. We have also added the missing algorithm for updating the archive as Algorithm 2 in Appendix A.

---

Q6: The reviewer had some questions about the meaning, definition, and impact of the hyperparameters p and q in the Equation 2, and whether they are adjusted adaptively.

A6: Sorry for the confusion. The hyperparameter q controls the weight of the measure bonus while p controls the relative impact of the different reward components. We use p and q simply because it is an interpretable linear function of the indicator function. These parameters are fixed throughout the optimization process. As our hyperparameter study shows (see Fig.10 in Appendix D.3 in the updated paper), our results can be further improved by tuning these hyperparameters. For your convenience, we summarize our results of hyperparameter study here:

The result shows that p=0.5 and q=1 is the best hyperparameter among our choices.

---

Q7: The reviewer was wondering whether the measure bonus can be applied to QD-RL and was interested in the Humanoid problem in this regard.

A7: This is a crucial point that we also considered during our research! Actually, the bonus can in principle be applied to QD-RL problem. We experimentally verify this on the Humanoid problem, and we now report results using PPGA with true reward function and measure bonus (see Fig.16 in Appendix J in the updated paper). For your convenience, we show the result table here:

The measure bonus can significantly improve PPGA for QD-RL problems. The rationales are similar: the measure bonus helps to mitigate the local optimal issue which severely inhibits exploring diverse behavior. In the QD-IL setting, this issue is exacerbated due to limited demonstration. We can also see that both with measure bonus, QD-IL can hardly outperform QD-RL.

Thank you for your time and valuable feedbacks. Here are our detailed responses.

---

Q1:This paper is the first to explicitly integrate standard QD algorithms with imitation learning, though it is not the first to study imitation learning methods that learns a diverse set of policies.

A1: Thank you for your response. The paper mentioned is indeed related to our setting of imitation learning from diverse demonstrations. However, the setting of that paper has a different aim to ours, namely to formulate one policy that is a function of behavior, while we are interested in designing a behaviorally diverse archive of high-performing policies. The authors have now updated the paper to refer to this cited work and we mention how the setting of this work differs from ours.

---

Q2: The reviewer has some questions about measure conditioning in terms of its empirical support to demonstrate how it helps to optimize diversity.

A2. Thanks for your valuable question. Actually, the measure conditioning is not intended to optimize diversity but rather to make the reward function more reliable, thus facilitating learning high-performing policies.

Here is the rationale: we observe that the measure of one state actually serves as a high-level abstraction of this state. Then, the expert can demonstrate how to behave when facing some specific type of state (where the “type” means the abstraction), instead of how to behave when facing some exact state. This is helpful because the state space is quite large, so it is hard to generalize the knowledge from limited exact state-action pairs.

For example, with the measure conditioning, the agent can learn to drop the right leg (action) when the left leg is not touching ground (which is reflected by measure, and also state abstraction). To improve the clarity, we have removed that sentence "enables the agent to learn broader behavior patterns”, and replaced it by “enables the agent to learn high-performing policies from limited demonstrations, improving generalization to unseen states”.

---

Q3: The reviewer mentions that there are other baselines, such as PWIL and GAIL, that also achieve much higher coverage than the PPGA expert in Humanoid, suggesting other factors affecting the coverage.

A3: That's a really great point! In general, different reward functions will have different optimization landscapes, which may in some cases lead to higher coverage for the PPGA algorithm. Actually, we have experimented using the zero reward for PPGA, and the result of coverage is approximately 100% but the QD score is extremely low, which is the same as PPGA combined with other reward based on PWIL and GAIL.

This phenomenon can be explained based on the inner mechanism of the PPGA algorithm. The PPGA algorithm’s objective is based on the CMA-MEGA algorithm, which is $g(\theta) = c_0 \cdot f(\theta) + c_1 \cdot m_1(\theta) + c_2 \cdot m_2(\theta) + \dots + c_k \cdot m_k(\theta)$, where $c_i$ are sampled from an evolutionary algorithm-based distribution. Please note that it is actually multi-objective, consisting of multiple terms. Optimizing one objective will potentially inhibit the optimization of another objective. Moreover, the first term $c_0 \cdot f(\theta)$ serves as the fitness (cumulative reward), measuring the performance of the policy, and other terms represent measures, representing the diversity of the policy. Hence, an extremely poor reward function (such as zero reward) will inhibit performance optimization but may facilitate diversity optimization. This leads to high coverage but a low QD score. On the other hand, a purely quality-based reward (such as the true reward function) will overemphasize performance and inhibit diversity optimization. This is exactly why we design the measure bonus.

---

Q1.: Thanks for added Table 9, but as the authors mentioned, out of the three tasks tested, measure conditioning only results in improvement of QD-Score that is beyond one standard deviation on one task.

Easy to fix: Currently Appendix E refers to Figure 7 for results in all three tasks, while Figure 7 only contains one task. Figure 11 seems to contain another task but is not referenced to. Similarly, the newly added Table 9 is also not referenced to by Appendix E.

A1. For the first part, indeed, due to the large variance in IL it is often difficult to guarantee large effect sizes. For the second part, we will take care to update the Figure references correctly.

Q2.: Section 5.3 mentions that "Each baseline learns a reward function, which is then used to train standard PPGA under identical settings for all baselines." I understand that the baseline methods all use PPGA as the QD-RL algorithm after a reward function is explicitly learned. My previous comment about the baselines learning a single reward function was about the stage before PPGA is applied.

A2. All the methods are learning a single reward function. Measure-conditioning is similar to learning with multiple reward functions but in this case the “intrinsic disadvantage” that the reviewer previously mentioned is simply concluding that our method works.

Q3.: In settings where longer episodes are indeed preferred for higher IL quality (which is likely true for the tasks used in the paper), the additional degree of positivity bias could be a factor that contributes positively in the policy performance, while the same positive effect can not be expected for IL tasks in general. This could make it more difficult to interpret some empirical results presented in the paper and compare the proposed method with the baselines.

A3. Thanks for your comments. To test this hypothesis, we have provided the result of MConbo-GAIL with $p=0$ and the default $q=0.5$ in Humanoid. The results show that the QD-Score of MConbo-GAIL ($p=0$ $q=0.5$) is still higher than all the baselines and is approximately equal to PPGA trained with True Reward. Therefore, the results do indeed generalize to $p=0$.