Discovering Creative Behaviors through DUPLEX: Diverse Universal Features for Policy Exploration

We appreciate your feedback, it has been very useful to strengthen our work.

(Ln 42-43) It seems to have a..

Humans adapt OOD contexts continuously. For example, if a person injures their leg, they can immediately balance on one leg and even jump forward without using the injured leg. We aim to transfer this adaptability to agents in different contexts. For instance, a bipedal robot learning to walk should explore various walking modalities to be ready for unseen situations. This example is included in Line 43.

(Ln 132) the paper claims…

We agree that it is not possible to empirically prove that DUPLEX finds a set of near-optimal diverse policies for every context. However, we do not claim this. Instead, our training objective is designed to promote configurations that maximize the expected return across diverse policies in each context. Specifically, we assert that DUPLEX finds diverse policies and generalizes better to OOD contexts than previous state-of-the-art methods while maximizing the training objective (Line 132). We revised Line 131 to state: “The objective of our algorithm is to find a set of diverse policies that maximize the expected return in every context.”

The Section 3 discussion...

Our goal is to find diverse policies that can be generalized to different contexts. We make use of function approximators to handle cases where it would not be possible to enumerate all possible contexts that an agent will encounter. We replaced line 118 with:

We briefly introduce the main building blocks of DUPLEX and the notation we use. First, we review basic concepts of multi-task reinforcement learning (RL) and explain how it can be represented by the contextual-MDP framework. Next, we describe the universal estimators used to enhance generalization across contexts, in cases where it would be hard to enumerate all possible contexts.

While I was able to follow Section 4…

We welcome the suggestion of the reviewer to improve clarity. We will lighten part of the preview at the beginning of Section 4.1.1 and introduce it at the beginning of Section 4 by describing how the different components are connected and interact.

The design of Equation 5 …

We explored various designs for Equation 5. Initially, we tried Lagrangian-constrained optimization but found it to be unstable, especially in domains like GT. Next, we tested a simple hard-threshold approach with \beta \dot v_{target}, but this made the reward signal discontinuous and more unstable. Ultimately, we used a sigmoid function to modulate the lambda parameter, as shown in Equation 5. This approach provided a continuous reward signal and effectively penalized deviations from near-optimality. Normalizing by |v_{target} + l| helped us avoid fine-tuning the sigmoid slope and maintain consistent hyperparameters across domains. We included this description in Appendix C.

The motivation for using an entropy…

Instability in RL often arises from predicting a moving target. Additionally, the complexity of training can increase depending on the application and task. We found that standard successor feature (SF) prediction is inadequate in multi-context settings.

We hypothesized that adding an entropy term to SF estimation helps the critic to account for domain uncertainty, similar to the motivation behind entropy in standard RL algorithms like SAC. This approach stabilizes estimation and prevents divergence, as shown in Figure 5. We further explore this in Appendix B and Figure 6, where we demonstrate that adding an entropy term also improves the standard USFA framework, achieving a 3x improvement in multi-context MuJoCo domains.

The features…

Definition 4.1 offers a general definition of policy diversity, aligning with related work [2] and it does not introduce a novel diversity metric (we clarified this in the paper). That said, to evaluate DUPLEX against other baselines, we avoided feature engineering and, in all MuJoCo experiments, used the entire observation vector for cumulants without selecting specific features.

In GT, we selected features that better characterize a driving style: action intensities (brake, throttle, and steering values), wheel angles, and distances to the track edges and cars. It is important to mention that in this type of domain, the observation vector includes hundreds of features and it might be difficult to tease out behaviors that can be visually appreciated to the naked eye. Initially, we did use the whole observation also in GT, and DUPLEX was indeed able to find much better diversity trade-offs, but, it was finding diversity also over less relevant features such as how much dirt there is on the tires, or how much tires are on tarmac and how many on the grass.

Minor comments…

We appreciate the reviewer's comments. While we acknowledge the limitations of SFs, we did not find them to hinder our ability to discover diverse behaviors. This is supported by the fact that our approach remains effective even when using averaged cumulant values along a trajectory. That said, it is an interesting future direction that could lead to improving the accuracy of our agent's predictions. As a first step, we could explore n-step returns when computing the prediction errors. We included this observation in the conclusions when discussing limitations.

Negative societal impact:

We will add: RL algorithms optimize decision-making in various domains. However, they also risk reinforcing biases and require careful implementation to ensure transparency and fairness. Diversity-learning algorithms might result in training policies with potentially unethical behaviors that are harder to predict. These are the reasons that motivate us to pursue controllability while learning diverse policies (Section 6, Question 4).

We thank the reviewer for their feedback, let us know if you have any further questions.

[1] Haarnoja, Tuomas et al. 2017. [2] Zahavy, Tom, et al. 2023.

We thank the reviewer for their interest in our work and the feedback provided. Below are the detailed responses:

Relying on multiple hyper-parameters (introduced by equations 3-5, probably hard to scale across the domain).

As indicated in lines 207-209 and 843-845, most of DUPLEX hyper-parameters do scale across domains used in our evaluations. As reported in Table 1, apart from network size and learning rate, to guarantee diversity, we only need to adjust \rho and \beta. It is worth mentioning that we want to expose such hyper-parameters to the user to be able to specify the performance vs. diversity trade-off according to the use case.

Some of the paragraphs has heavy notations and might be more readable if the author could give more intuitions on why it came out like that. For example in section 4.1.1 and section 4.2.

Reviewer XDei expressed a similar comment and suggested adding a preview paragraph describing how the components interact with each other at the beginning of Section 4. We, therefore, redirect this comment to that answer as well.

Regarding the diversity definition in Def 4.1, can you give an example of whether/how different modalities of observed features (e.g., visual or textual) could have a different impact on the diversity formula? or it does not matter (no effect).

This is an interesting observation. We believe that different modalities do not alter Definition 4.1, which holds valid. We can always compute L2-norm distances of tensors.

However, under different modalities, it is important to understand how to design cumulants and guarantee that the agents focus on important features. Similar images can represent different things and the same concept can be expressed with very different words. For example, in GT, few pixels can draw the line between a mispredicted collision and an actual collision between cars.

One approach is to preprocess visual and textual information with an encoder, or we could design modality-invariant cumulants. Nevertheless, guaranteeing execution under different modalities is an interesting aspect that is mandatory to generalize DUPLEX to diverse domains, and will be a topic in our future work.

What is the relative additional cost introduced by computing SF distance of each policy pair? We will compute the execution time and report the additional cost of estimating SFs per batch iteration. It will be included in Section D. The computation of the distance of all the samples in the batch is done through matrix multiplication, not iteratively.

I like how the author clarified their limitations in section 6. I am very interested in learning more explanations on questions #3 and #4.

We appreciate your comment and your interest in our future work. Below we detail questions 3 and 4:

Q.3) DUPLEX does not impose any exploration strategy on each policy …

By design, DUPLEX does not bias exploration within the near-optimal regions and policies can converge to any local-minima in such space as long as the diversity constraint is satisfied. As a result, two consecutive experiments within the same domain may return with a different set of diverse policies. However, for some use-cases it is reasonable to guarantee that some behavior is always discovered. For instance, in Walker2D, there is always a policy that learns to jump by only using the left leg. In GT, an aggressive policy is often learned. In future work, we could explore how to differently mask cumulants for each policy in the set and bias exploration. Alternatively, we could design SF vectors that represent desired behaviors and use such vectors as behavior baselines for each policy.

Q.4) can we combine the strengths of different on a single solution? …

Intuitively, even though the target policy is configured to only maximize the extrinsic rewards, the other policies (due to the added exploration) could converge to behaviors that are more efficient in specific state transitions. We believe that enhancing the target policy with situational corrections coming from the auxiliary policies can improve the overall performance of the agent.

In early experiments, we explored generalized policy improvement (GPI) [2], which intuitively, uses the critic estimations to select, at each step, the best action over a set of policies. However, empirically, we did not find it to improve the performance of our agent. We hypothesize that the imperfect information of the critic becomes especially relevant in continuous-action environments. Moreover, our continuous settings might demand GPI to be executed with a longer horizon, and a “simple” step-wise composition of diverse policies might not be sufficient.

Finally, we believe that improving accuracy in the critic estimations and taking advantage of the added exploration of diverse learning approaches is an exciting line of work and is necessary to advance the research field.

[1] Wurman, Peter R., et al. "Outracing champion Gran Turismo drivers with deep reinforcement learning." Nature 602.7896 (2022): 223-228. [2] Barreto, Andre, et al. "Transfer in deep reinforcement learning using successor features and generalized policy improvement." International Conference on Machine Learning. PMLR, 2018.

We thank the reviewer for the time and appreciate the feedback provided.

There is a strong assumption of the method: the context vector is given by the environment.

We agree that not all environments are designed to provide a context vector. However, we comply with related work in zero-shot generalization [1] where contextual information is provided as a separate input that is procedurally generated by the environment. That said, we believe that extending our work to domains with hidden contextual information is a promising future direction and will include it in the conclusion section. However, it is out of the scope of this paper.

[1] Kirk, Robert, et al. "A survey of zero-shot generalization in deep reinforcement learning." Journal of Artificial Intelligence Research 76 (2023): 201-264.

The claim on context-conditioned MDP…

We agree that only extending Equation 2 to contextual-MDPs is not a significant contribution and thus it is not listed as a main contribution. To have a fair comparison, we also provide contextual information to DOMiNO when evaluating it.

To highlight the novelty, we list the technical contributions briefly here. These are detailed in Section 4 and the ablation studies. Dynamic intrinsic reward factor to modulate the ratio between diversity and extrinsic rewards, alleviating the reward tuning that DOMiNO requires across different environments Soft-lower bound to limit the region of interest for the diverse policies. Entropy regularisation term in SFs estimation to improve the estimation of SFs in multi-task environments. Using the average of the critic estimates is beneficial when estimating SFs and at the same time computing diversity

To further highlight the novel components that DUPLEX introduces, we added a brief paragraph summarizing key contributions at the end of Section 4. As evidenced in the ablations, all these components have a significant impact with respect to DOMiNO and enable our agent to achieve competitive performance in hyper-realistic domains and in multi-task environments.

The method is not scalable…

Please note that Equation 2 and Equation 3 are not computed separately for each pair of policies but instead are vectorized through matrix multiplication, and thus scalable. We note that the equations are computationally equivalent to the equations in Zahavy’s method (which provides pseudocode on how to compute these equations as a matrix). We specified in Appendix C that we do not increase computational complexity with respect to previous baselines.

No code is provided.

We cannot share the code to this date. We included a detailed pseudo-code description of our algorithm, described our environment domains, and included the hyper-parameters used for training DUPLEX agents. We believe this to be the key information needed to reproduce our experiments but we are happy to include further details otherwise.

What does "closest expected features" …

According to Definition 4.1, we compute distances among policies as the L2-norm distance between their feature vectors. We added a reference to Definition 4.1 on line 178.

Clarity issues

We thank the reviewer for the detailed review. We have integrated the recommended corrections into the equations. Note that the definition of y follows in Equation 8 and we have updated the text to better describe this connection.

What does the braces in Eq 5 means?

Braces indicate that the soft-lower bound is defined as a vector of n = size(\Pi) components, one for each policy. We corrected a typo in the equation that as currently defined, would consider n + 1 policies.

How ϕ(s,a,c) is computed? The …

We compute expected features ψ by predicting the expectation of the cumulants ϕ through the estimation of the critic network. And we achieve that by adding an extra head to the critic network.

More specifically, cumulants ϕ are a part of the input vector where we want to maximize diversity. By default, we use the whole environment observation as ϕ. Nevertheless, if we want to tease out diversity in particular features, as in the case of GT, we only use of subset of the observations to define ϕ. For example, in the aggressiveness scenario shown here https://sites.google.com/view/duplex-submission-2436/home, only the part of the observation that characterizes collisions among cars is included in ϕ, and as a result, the agent trains policies with different levels of aggressiveness.

Expected features ψ are directly estimated by an additional head of the critic network (Line 8) and the error is obtained through Equation 7. The critic architecture is detailed in Figure 1. and the size of the expected features vector is equal to the number of cumulants. We included this description in Table 1.

It seems the method use QRSAC…

We use SAC in the canonical MuJoCo environment for repeatability of the results and use QRSAC in GT as it represents the state-of-the-art in that domain.

Since the intrinsic reward depends on the transitions sampled from the environment, we compute the reward signal for each batch. Nevertheless, this operation is vectorized through matrix multiplication and it is not iteratively computed, which would make the approach impractical.

Missing citations to relevant works

We are happy to include these references, we also think that exploration is key when learning diversity and these citations represent a promising future direction to further improve our method.

We appreciate the time and feedback you provided on our submission.

The paper does not compare its diversity objective to other existing diversity objectives in the literature, such as DIAYN (Eysenbach et al., 2019) and MaxEnt RL (Haarnoja et al., 2018). This omission makes it difficult to judge why this particular diversity measure should be considered over the others.

Including algorithmic baselines can greatly enhance a scientific contribution. For our experimental section, we selected baselines that we believe best support the claims of our work. We consider DOMiNO to be the state-of-the-art method in reward-based diversity learning and thus encompasses other baselines. In contrast to DIAYN and MaxEnt RL (and DOMiNO), DUPLEX is also being evaluated in OOD multi-task generalization. In such a setting it would not be a valid data point for the comparison against standard diversity learning algorithms. To support the contribution of DUPLEX in such settings, we rely on state-of-the-art frameworks for multi-task learning such as UVFA and USFA.

The paper builds upon the Domino paper, but it is not clear how it differs from Domino since both papers use extrinsic rewards and another metric computed to promote diversity, i.e., intrinsic reward.

As the reviewer correctly points out, DUPLEX’s core definition of diversity and diversity reward computation is aligned with DOMiNO. However, we extended the training problem to include context-based scenarios and enhanced the robustness of training diverse policies. We achieved this by adding new components, which are briefly outlined here (and detailed Section 4):

• Dynamic intrinsic reward factor to modulate the ratio between diversity and extrinsic rewards, alleviating the reward tuning that DOMiNO requires across different environments
• Soft-lower bound to limit the region of interest for the diverse policies.
• Entropy regularization term in SFs estimation to improve the estimation of SFs in multi-task environments.
• Use of the average of the critic estimates. Differently than taking the min of the critics as it is common with value estimation, we find that to compute diversity rewards is beneficial to take the average of the estimated SFs.

To further highlight the novel components that DUPLEX introduces, we added a brief paragraph summarizing key contributions at the end of Section 4.

Another fundamental question arises: If a model can generate stochastic policies with high rewards, will it be considered diverse compared to the deterministic policy model? Is diversity essentially a measure of generating high-reward stochastic policies?

In this work, according to Definition 4.1. we consider (deterministic and stochastic) policies to be “diverse” if they feature diverse state-action occupancies ). A stochastic policy will visit different state-action pairs than a deterministic one. Hence these two policies can be considered diverse. When introducing a near-optimal constraint, we are still searching for policies that feature diverse state-action occupancies, but we constrain the search within a particular area of the search-space, which is defined by Equation 1.

In other words, the reward obtained by the agent (or model) has no impact on the computation of the diversity metric, but since DUPLEX enforces that all the policies have to simultaneously optimize for the true objective as well, it yields the ability to maximize discounted cumulative rewards (high-rewards).

I would urge authors to upload code and some visualizations with comparison of the method to Domino during rebuttal period.

We cannot share the code to this date. We included a detailed pseudo-code description of our algorithm, described our environment domains, and included the hyper-parameters used for training DUPLEX agents. We believe this to be the key information needed to reproduce our experiments but we are happy to include further details otherwise.

We are working towards including visualizations of Domino vs DUPLEX in our site https://sites.google.com/view/duplex-submission-2436/home but, we will likely add them after the discussion period ends.

In the meantime, we can describe our observations from completed experiments. DOMiNO typically produces policies with subtle differences, as seen in Figure 1(b) of [2], where each policy raises the leg slightly more than the previous one. Occasionally, by sufficiently reducing the optimality ratio, we may observe two “main behaviors” that are visually distinguishable, with other policies showing small variations around them. In contrast, DUPLEX tends to produce significantly more diverse populations of near-optimal policies that are visually distinct. This greater diversity yields better trade-offs, as shown in Figures 2, 3, and 4.

Thank you for your useful feedback. We will be happy to respond to any further questions you might have.

[1] Zahavy, Tom, et al. "Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality." The Eleventh International Conference on Learning Representations.

We appreciate the feedback you provided on our submission.

In 4.1.1, the authors …

Please note that our results conform to the literature [2, 3, DOMiNO] when evaluating the Lagragian-constrained optimization in canonical MuJoCo environments and we found it to outperform the non-lagrangian counterparts. However, when considering a complex domain like Gran Turismo, where the difference between a competitive and an unsatisfactory policy could be a few hundred milliseconds (when evaluating the lap time in completing a track) and state transitions obey complex dynamic interactions, we found that Lagrangian multipliers are not as beneficial. Also, note that this kind of domain is not considered in [1, 2] or in DOMiNO. In complex domains, we report that Lagragian-constrained optimization fails to preserve a good performance vs diversity trade-off. This is evident in Figure (2, a) where DOMiNO is only optimizing for performance, disregarding the diversity component of the training objective. Mathematically, Lagragian-constrained optimization is known to be unstable [4] while searching for a good compromise across different constraints. When evaluating Lagragian-constrained optimization in DUPLEX, we found that rapid changes in the multipliers are not beneficial when learning complex environmental dynamics and the agent would either optimize for performance or diversity. This observation led us to introduce the soft-lower bound in DUPLEX – which as we report in Figure (2, b-c) outperforms other baselines.

In the experimental section...

We welcome the reviewer's suggestion and we are working towards including the diversity score of a vanilla SAC baseline in our experimental section. It is worth noting that we are normalizing diversity scores with respect to the most diverse set of policies across domains to facilitate comparison. The reported increment is then a squashed score but, as we report in Figure 3., it is important to note that DUPLEX can provide a more stable performance vs. diversity trade-off. While we cannot provide the requested baseline yet, we can share data demonstrating the correctness of the DOMiNO implementation:

• In Figure 3, the average reward and mean diversity of DOMiNO are presented for two different optimality ratios (OR). With an OR of 0.9, DOMiNO achieves an average reward of 6140 ± 94.34 and a mean diversity of 0.0215 ± 0.0086. At an OR of 0.5, the average reward is 5568 ± 698.70, with a mean diversity of 0.0372 ± 0.029. For comparison, we evaluated USFA in a single-task Walker scenario. USFA, which uses SAC and lacks diversity incentives, achieved an average reward of 6280 ± 555 and a mean diversity of 0.003 ± 0.001. USFA results are also based on 5 independent runs. These findings indicate that DOMiNO produces policies that are 7 to 12 times more diverse than those of USFA. The plot normalizes these differences, which appear minimal when compared with DUPLEX.
• In our code-base, both DOMiNO and DUPLEX use the same utility functions; core computation of the intrinsics reward; Lagrange-constrained optimization, and SAC as the base algorithm.
• Figure 4 illustrates the performance of different methods on a multi-context benchmark. As mentioned in Section 4.2, with infinite environments, we cannot use the average of inputs to compute Successor Features (SFs) and must estimate them instead. This introduces instability, causing both SFs and diversity rewards to chase correlated errors. DOMiNO lacks the stabilizing mechanisms introduced with DUPLEX, resulting in its failure to learn competitive and diverse policies, similar to USFA. Figure 5 highlights the importance of these mechanisms, showing that without them, DUPLEX would also fail. Appendix B demonstrates that incorporating entropy regularization into USFA (as in DUPLEX) significantly improves its results.

Minor corrections

Thank you for the pointer, we integrated these corrections to the main body of the paper.

Code.

We cannot share the code to this date. We included a detailed pseudo-code description of our algorithm, described our environment domains, and included the hyper-parameters used for training DUPLEX agents. We believe this to be the key information needed to reproduce our experiments but we are happy to include further details otherwise.

Lastly, as a reader…

The approach presented in [3] exploits policy distillation to extrapolate common skills from strategies learned across the different tasks. As pointed out by the reviewer, the diversity score instead serves a different purpose. That is, in contrast to [3], our training objective is based on encouraging diversity subject to the performance of the target policy. Such diversity is especially important in generalizing to OOD tasks, as illustrated in Figure 4 b). This is because policies can converge to strategies that share fundamental skills but learn different strategies that can be useful to unseen tasks. For example, in our Walker2D environment with changing gravity, a policy that applies higher torques to the joints has a higher chance to succeed under higher gravitational force conditions.

Finally, demonstrating near-optimal performance while combining diversity and multitask generalization has direct applications to the GT player community, increasing replayability and engagement for users by racing against diverse and competitive opponents. Being robust towards OOD tasks and environments makes these agents more applicable to new game functionalities and updates.

Thank you for your feedback. Let us know if you have any further questions.

[4] Moskovitz, Ted, et al. "Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps." International Conference on Machine Learning. PMLR, 2023.