We thank reviewer XcFd for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q1: The phrase 'out of the box' in the title is not clearly connected to specific concepts or points within the paper. The authors should clarify what 'box' refers to in the context of their method.

A1: The phrase "out of the box" metaphorically refers to the capability to transcend the typical constraints posed by the training dataset in offline reinforcement learning. In the conventional setup, RL agents are often strictly to learn actions within the dataset. Our method, however, is designed to enable learning behaviors outside the dataset. The box implies the data constraint. We are happy to rephrase the title if the phrase "out of the box" is not explicit.

Q2: The paper states, "Agents are rewarded for behaving differently from each other during training, providing direct guidance that allows policies to learn behaviors even beyond the dataset distribution." Why does the proposed method enable learning behaviors beyond the dataset distribution? How does this approach differ from importance sampling-based methods, and why did the authors choose not to use importance sampling?

A2: The key to enabling behaviors beyond the dataset distribution lies in the design, where policies are rewarded for behaving differently from each other during training. Offline RL is not a mere behavior cloning; rather, policies are guided through value functions that assess and update model parameters based on the policy's actions. Under offline constraints, the value function is trained solely on pre-collected data. However, when policies are encouraged to maintain distinct action distributions, they may naturally be pushed beyond the dataset's scope. This approach fundamentally contrasts with importance sampling, which continues to sample from the original dataset distribution and does not facilitate policy actions outside of the dataset range.

Q3. What distinguishes the current mutual information-based approach from other methods, including the baseline approaches like DIVEOFF, CLUE, and SORL? The authors mention that DIVEOFF also maximizes mutual information, so why does their method achieve better diversity while balancing performance? What is the underlying reason for this difference?

A3: Our approach and DIVEOFF differ fundamentally in the types of actions they focus on for mutual information (MI) maximization, leading to distinct outcomes in terms of policy behavior and diversity. Our method maximizes MI between the policy and the actions it selects during training. These actions are generated by the policy itself, which means they are not restricted to existing state-action pairs within the dataset. The only constraint is that different policies should exhibit noticeable behavioral diversity, thereby encouraging exploration even beyond the dataset distribution. In contrast, DIVEOFF, along with other baseline methods like CLUE and SORL, focuses on actions within the dataset, specifically the state-action pairs pre-collected in the training data. DIVEOFF maximizes MI between latent vectors and these state-action pairs, inherently limiting the policy’s behavior to what the dataset already contains. By emphasizing MI in the policy’s own selected actions, our approach achieves diversity beyond the dataset. EDAC further ensures that this exploration is done with reliable value estimates, enabling the policy to balance performance and diversity effectively.

Q4. Why does the approach adopt an ensemble method (EDAC) to avoid over-estimation? How does this compare to other techniques like conservative Q-learning or conservative policy gradient methods? What are the advantages of using EDAC over these alternatives?

A4: In offline RL, mitigating action value over-estimation is crucial, with two primary strategies often applied: conservative methods and uncertainty awareness.

Conservative Methods: Techniques like Conservative Q-Learning (CQL) and TD3+BC focus on guiding the agent to favor actions within the dataset, thereby minimizing over-estimated value predictions and stabilizing learned policies. However, the conservative approach limits the exploration of novel actions outside the training data, which may hinder objectives related to diversity and exploratory behaviors.

Uncertainty Awareness with EDAC: Methods such as Ensemble Distributional Actor-Critic (EDAC) use ensemble models to quantify uncertainty in value estimations. When the agent encounters actions outside the dataset, the ensemble provides multiple value estimates, enabling a more robust average and variance calculation. This ensemble-based approach allows the policy to better assess the potential risks and rewards associated with OOD actions, providing more reliable estimations even when extrapolating beyond the dataset. By focusing on actions with lower uncertainty, EDAC naturally enables policies to explore a broader action space while controlling for over-estimation, aligning with our goal of fostering diversity without compromising performance.

Q5. In Algorithm 1, how is the policy update in line 8 implemented? Did you use the policy gradient theorem? The authors should provide more details on how the policies are updated.

A5: In Algorithm 1, the policy update mechanism employed in line 8 is based on the framework used in Soft Actor-Critic (SAC), with a key modification tailored to enhance robustness and stability. Specifically, while SAC typically updates policies using a standard policy gradient method, we select the Q-function that provides the minimum value for the current policy from among those available in an ensemble of Q-functions (EDAC). This selection criterion is promoting more stable learning by mitigating the risk of overestimation in offline RL, particularly when policy's actions are outside the dataset.

Q6. The paper states, "However, a challenge arises when the dataset is homogeneous, or when distant latent vectors are decoded into similar actions." What exactly does it mean for distant vectors to be decoded into similar actions? Could the authors provide a more detailed explanation of this concept?

A6: A decoder taking distinct inputs is not guaranteed to generate different outcomes. This issue is analogous to mode collapse in generative models, where diverse inputs (latent vectors) lead to indistinguishable outputs, failing to capture the potential diversity the model is theoretically capable of expressing. Latent conditioned policies, like those employed in DIVEOFF, map latent vectors to actions within a dataset, and likely suffer from the same problem. This phenomenon is evident in the results for DIVEOFF, as depicted in Figure 1, where different latent vectors yield similar policy behaviors.

Q7. Regarding the abstract, after reviewing the full paper, I found that the comparison between heterogeneous and less diverse datasets is not explicitly addressed in the methodology or experiments. The claimed effectiveness of the approach across different data types has not been adequately substantiated. Could the authors specify which datasets are considered heterogeneous and which are less diverse, and clarify this distinction in the methodology or experiment sections?

A7: Standard D4RL Dataset is generated using a Soft Actor-Critic (SAC) policy, which typically results in homogeneous behavior patterns. In contrast, the Diverse D4RL Dataset is collected using a latent-conditioned policy that is specifically designed to capture a wider range of behaviors, making it more heterogeneous. The Diverse D4RL Dataset exhibits greater diversity compared to the Standard D4RL Dataset due to the inclusion of multiple latent-conditioned policies. We apologize for not providing a detailed description of the Diverse D4RL Dataset earlier, as the dataset was collected by the authors of SORL. We will revise the manuscript to more clearly distinguish between the Standard and Diverse D4RL datasets, and to clarify their differences in both the methodology and experiments sections.

Q11. The estimation formula $p(a_t | s_t) = \sum_{i=1}^M p(a_t | s_t, \pi^i) p(\pi^i)$ on page 5 is intriguing, but can it truly provide an accurate estimate of $p(a_t∣s_t)$? Are there any potential issues with bias or high variance in this approach? The same concern applies to the formula $p^t(s_t) = \sum_{i=1}^M p^t(s_t | \pi^i) p(\pi^i)$. Could the authors explain the rationale behind using these estimations? If this method is derived from existing research, it would be helpful to cite those sources and provide further clarification.

A11: In our unique behavior approach, we focus exclusively on computing the value $p(a_t | s_t)$, which is directly determined from the available policy outputs. Importantly, our method does not require the estimation of $p_t(s_t)$, the state transition probability, which simplifies the computational process and reduces the potential for introducing estimation errors.

Specifically, In the formula $p(a_t | s_t) = \sum_{i=1}^{M} p(a_t | s_t, \pi^i) p(\pi^i)$ the term $p(a_t | s_t, \pi^i)$ is defined as $\pi^i(a_t | s_t)$, where $\pi^i$ represents the policy model's output for policy $i$. Without loss of generality, we assume a uniform distribution over the policies where $p(\pi^i) = \frac{1}{M}$, for each $i$. As we have full knowledge of each policy $\pi^i(a_t | s_t)$, there is no need to estimate these probabilities. This direct utilization avoids the introduction of additional bias or variance that often accompanies estimation processes.

Q12. In Inequation (8), is the second term of $U_{\text{Path}}^{\pi^i}(\tau) = \sum_{t=0}^{T} \log \left( \frac{p(a_t | s_t, \pi^i)}{p(a_t | s_t)} \right) + \sum_{t=0}^{T} \log \left( \frac{p^t(s_t | \pi^i)}{p^t(s_t)} \right)$ always positive? It is not clear to me why this inequation holds. Could the authors provide further explanation?

A12: Adapting the mutual information concept from Equation 4 to the context between the state $\mathcal{S}$ and the policy $\Pi$, we have: $I(\mathcal{S}; \Pi) = H(\mathcal{S}) - H(\mathcal{S}|\Pi) = \mathbb{E}_{(\pi, s) \sim p(\Pi, \mathcal{S})} \left[ \log \frac{p(s|\pi)}{p(s)} \right]$ Given the non-negative nature of mutual information, this formulation supports the validity of Inequation (8).

Q13. To which part of the paper does Figure 2 correspond? The authors did not clearly indicate this connection.

A13: Thank you for pointing out the omission. Figure 2 corresponds to the discussion on lines 284 to 293 of the paper. We have revised the document as: In Figure 2, we use a 2D maze environment to illustrate how increasing behavior diversity can lead to greater path diversity. In this environment, the corridors are shown in white and the walls in gray. The goal is for agents to navigate from various starting positions to a target located at the bottom-left corner. Agents only receive rewards when they are within a 0.5-unit radius of the goal, which is visually indicated by a circle. The dataset consists of segments $(s, a, s')$, where $s$ is the current state, $a$ is the action taken, and $s'$ is the subsequent state. By maximizing behavior diversity, the experiment showed that agents followed different routes to reach the destination. This outcome is expected, as a trajectory is a sequence of states and actions, and small differences in local states accumulate over the length of the trajectory, leading to distinct paths.

Q14. The notation $\phi(\pi_i) = E_{s \sim \pi_i, P}[s]$ is used to evaluate state diversity, while $\phi(\pi_i) =E_{s \sim \pi_i, P} [a]$ is used to assess action diversity. However, it appears that the same notation $\phi(\pi_i)$ is being used for two different values. Could the authors clarify this?

A14: In our formulation, $\phi(\pi_i)$ serves as a generic notation for a behavioral embedding function within the kernel calculation. This function is adapted based on the context—either state or action diversity. To alleviate any confusion and improve the clarity of our methodology, we redefine the notation to reflect these distinct applications more clearly: $\phi_s(\pi_i) = E_{s \sim \pi_i, P} [s]$ to evaluate state diversity and $\phi_a(\pi_i) = E_{s \sim \pi_i, P} [a]$ to assess action diversity. We have revised the document and have uploaded the new PDF version. Please refer to this latest version for the updated content.

Q15. Is this method applicable only to offline learning? If not, please consider discussing its potential for online scenarios in the future work section.

A15: While our current implementation and evaluations primarily focus on offline learning scenarios, the underlying principles of our method have broader applicability, including potential extensions to online learning environments. The core principle of our approach—encouraging unique behavior among agents—remains effective regardless of the learning context. This is because the incentive for agents to adopt diverse and unique actions is not inherently linked to the data being static or dynamically generated. In online learning, where agents interact with an environment and decisions are immediately reflected in the environment, the same intrinsic rewards designed to promote behavioral diversity can be employed. These rewards would guide the agent to explore novel actions during its interaction with the environment, enhancing its ability to adapt to evolving situations and potentially improving its exploration strategies.

Q8. How important is diversity for reinforcement learning, particularly in offline RL? What are the main benefits of promoting diversity in this context? The authors should elaborate more on the significance and implications of this aspect in their research.

A8: Below, we discuss the advantages of diversity for RL in different aspects.

Novelty and Engagement: In applications such as conversational agents, diversity ensures that interactions remain engaging and novel, even across many sessions with the same user. This can prevent conversational stagnation and maintain user interest and satisfaction over time.

Preference and Personalization: Diversity allows for the development of personalized policies that can cater to specific preferences or requirements. In scenarios like recommendation systems or adaptive interfaces, where user satisfaction is paramount, having a diverse array of strategies enables the system to tailor its responses more effectively to individual users.

Simulations: In simulations involving groups of agents, such as in crowd simulation or multi-agent systems, diversity can lead to more realistic and high-fidelity models of group behavior.

From the perspective of applications, diversity is equally important in online and offline RL since they differ mainly in training environments.

Q9. In Table 4, the performance of the EDAC method appears to be better than that of the proposed approach. Under what conditions should we choose the proposed method for its diversity benefits, despite the lower performance compared to EDAC? Is there a standard metric or conventional score to evaluate this trade-off between diversity and performance?

A9: More than performance: While the EDAC method may exhibit better performance on certain benchmarks, our proposed method offers unique advantages in scenarios where diversity in agent behavior is critical. This is particularly important in applications requiring human-agent interaction or multi-agent environments. For example, in interactive applications such as gaming or conversational agents, diverse responses or actions can enhance user engagement, ensuring sustained interest and participation over time. In such contexts, the benefits of diversity outweigh the marginal differences in performance.

Evaluating Trade-offs: As for evaluating the trade-off between diversity and performance, we acknowledge that there is currently no widely accepted standard metric for this purpose. However, we address this trade-off in our analysis (Figure 1), which visually represents the relationship between these factors. Ideally, the most desirable policies are those situated in the upper-right quadrant, demonstrating both high diversity and strong performance.

Q10. I noticed that the proposed method appears to outperform DIVEOFF, SORL, and CLUE in both performance and diversity. Could the authors also evaluate the diversity of other high-performance algorithms, such as EDAC, CQL, and IQL? Furthermore, are the observed differences in actions genuinely due to enhanced diversity, or could they be attributed to insufficient training of the policies (e.g., SORL), which may have been limited to a single, suboptimal policy?

A10: Algorithms like EDAC, CQL, and IQL typically train a single policy. For a given input state $s$, the action produced by these policies remains identical regardless of how many times they encounter $s$, resulting in a diversity score of zero. For evaluation, DIVEOFF’s authors modify IQL by integrating it with a Variational Autoencoder (VAE) to create a latent-conditioned version (IQL+VAE). As reported in their study, DIVEOFF achieves higher diversity compared to IQL+VAE. Our method further surpasses DIVEOFF, demonstrating superior performance and diversity.

For policies trained using EDAC, CQL, and IQL, insufficient training would result in poor decisions overall but would not affect the deterministic nature of their actions for a given state $s$. In contrast, methods such as SORL, DIVEOFF, CLUE, and our proposed approach produce varying behaviors for the same state $s$ due to the use of latent vectors or policy indices. These differences in behavior are intrinsic to the model's design and cannot be attributed to insufficient training. Instead, they reflect the intended diversity mechanisms embedded within these methods.

We thank reviewer BSoj for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q1. This paper proposes a new intrinsic reward system that maximizes the mutual information between actions and policies, encouraging behavioral diversity. Through the combination of EDAC method to estimate Q-value uncertainty, the proposed method achieves better performance than other baselines. However, the main concern is that the idea of introducing mutual information in offline RL is also proposed by one previous work [1]. Eq. 5 and 6 of [1] are similar to the objective function in this paper. The authors should claim the difference and novelty compared to [1] and consider involving it as a baseline in experiments.

A1: In our model, we explicitly consider a multiple-policies model. This multi-policy perspective allows us to investigate the interactions and behavioral variances among different policies, which is critical for enhancing exploration beyond the dataset's confines. This stands in contrast to the approach taken by Ma, Xiao, et al., where the framework is built around a single-policy model. Their primary objective is to regularize the agent within the scope of existing data, focusing on reducing the uncertainty of actions given specific states without the need to account for interactions between different agent strategies. The two approaches also differ in their algorithmic formulations:

• Ma, Xiao, et al.'s Model: They define the mutual information between the state $S$ and action $A$ as: $I(S; A) = E_{p(s, a)}\left[ \log \frac{p(s, a)}{p(s)p(a)} \right] = E_{p(s, a)}\left[ \log \frac{p(a \mid s)}{p(a)} \right]$ focusing on minimizing the uncertainty of actions in given states, thereby aligning closely with historical data to reduce risks.
• Our Approach: We define the mutual information between action $A$ and policy $\Pi$ in the context of state $S$ as: $I(A ; \Pi | S) = E_{(a, \pi) \sim p(A, \Pi)} \left[ \log \frac{p(a | s, \pi)}{\sum_{i=1}^M p(a | s, \pi^i)p(\pi^i)} \right],$ where our focus is on maximizing the diversity among actions taken by different policies, highlighting the unique contributions each policy can offer to the decision-making process.

Given these substantial differences in focus, methodology, and objectives, our work advances the domain of offline RL by exploring new avenues for action diversity and policy interaction. We acknowledge the foundational contributions of Ma, Xiao, et al. and will consider incorporating their model as a baseline in future experiments to further delineate the unique aspects of our approach.

Dear reviewer BSoj,

We appreciate the time and effort you have invested in reviewing our manuscript. If possible, could you kindly let us know if our responses have adequately addressed your questions? Please let us know if you have any further comments. We would be delighted to provide additional responses. Thank you.

We thank reviewer 8Awq for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q1. How are the heterogeneity and homogeneity of the dataset defined in the context of this study?

A1: In the context of our study, heterogeneity in a dataset is defined by the variety of user behaviors it contains. For example, in driving scenarios, heterogeneity is observed through diverse driving styles, such as some drivers preferring to closely follow the car ahead, while others frequently change lanes. This variance exemplifies heterogeneity. Conversely, homogeneity refers to the uniformity in user behaviors within the dataset, where most data points exhibit similar actions or decision-making patterns, often seen in scenarios where drivers show consistent driving styles. In our experiment, the Standard D4RL Dataset is derived from a Soft Actor-Critic (SAC) policy which generates homogeneous behavior patterns. In contrast, the Diverse D4RL Dataset is collected from a latent-conditioned policy, with various latent vectors, designed to capture a broader range of behaviors, hence more heterogeneous. As illustrated in Table 1, the baseline methods (DIVEOFF, CLUE, SORL) trained on heterogeneous datasets received high diversity scores; but their diversity dropped significantly when they were trained on homogeneous datasets. In contrast, our method achieved high diversity in both homogeneous and heterogeneous datasets.

Q2. If I understand correctly, In Line 205-208, $p^t(s_t|\pi^i)$ is $p(s_t|s_{t−1},a_{t−1})$ and is nothing to do with $\Pi$ (since you have specified $\tau$ in $p(\tau|\pi)$, it can decompose to transition probability $p(s_t|s_{t−1},a_{t−1})$ and $p(a_t|s_t,\pi)$for t=0,1,...). Based on this, I think there may be some mistakes in the following analysis.

A2: We would like to clarify our notation regarding state transition probabilities. In our framework, $p^t(s_t | \pi^i)$ denotes the probability of agent $\pi^i$ being in state $s_t$ at time $t$, conditioned on the entire sequence of states and actions from the start up to time $t-1$: $s_0, a_0, \ldots, s_{t-1}, a_{t-1}$. This formulation allows for a more general expression than the typical one-step transition probability: $p^t(s_t | \pi^i) = p(s_t | s_{t-1}, a_{t-1}, \ldots, s_{t-k}, a_{t-k})$ where $k$ ranges from 2 to $t$, depending on the depth of historical dependency considered. Notably, in the standard one-step Markov Decision Process (MDP) setting, this reduces to: $p(s_t | s_{t-1}, a_{t-1})$ and thus upholds Equation 8. Our formulation also holds in $k$-steps MDP settings, accommodating dependencies over multiple previous steps and also other historical conditioned transition probability. Therefore, our formula is applicable in scenarios involving more complex historical dependencies beyond the MDP framework.

Q3. In practice, how to estimate p(at|st) in Equation 8?

A3: Since $p(a_t | s_t, \pi^i) = \pi^i(a_t | s_t)$, we have $p(a_t | s_t) = \frac{1}{M} \sum_{i=1}^M \pi^i(a_t | s_t)$, where $\pi^i(\cdot | s_t)$ is the agent $\pi^i$ and under the presumption of $p(\pi^i) = \frac{1}{M}$ . Hence, $p(a_t|s_t)$ can be obtained exactly through forward passes of the policy networks.

Q4. Can the experimental results be generalized to tasks with discrete action spaces, such as Atari, representing more complex environments? It would be advantageous if the methods could be validated more broadly.

A4: We extended our evaluation to the Atari domain, where our comparison is primarily with SORL, as it is the only baseline method in our study that has also been tested on Atari environments. We executed the official SORL code available on their GitHub repository to ensure a fair comparison. For this part of our study, we set $\lambda$( the weight for $U_{\text{Behavior}}$) to 1 We trained a model with three different policies and tested each across 10 trajectories using three random seeds to assess variability and consistency in performance. The results from these experiments indicate that our method not only performs well across various conditions in MuJoCo tasks but also extends effectively to other environments, including those with discrete action spaces, such as Atari.

Q5. The authors employ ensemble training in the experiments. Could we consider incorporating information between the policies to achieve final diversity, as done in [1]? [1] Wu S et al. Quality-similar diversity via population based reinforcement learning[C]//The Eleventh International Conference on Learning Representations. 2023.

A5: In Wu et al. In [1], diversity is engineered through the use of user-specified Behavior Descriptors, which promote varying agent behaviors to align with different user-defined criteria. In contrast, we use unique behavior (UB) to measure how unique the action is among agents, which allows for a more intrinsic exploration of diversity. However, integrating the concept from Wu et al. [1] into our framework could be beneficial for scenarios where specific types of diversity are desirable. By adopting their strategy of using Behavior Descriptors, we can tailor the diversity generated by our model to fit particular user needs or to ensure alignment with targeted goals.

We conducted experiments (please refer to: https://imgur.com/a/e0TKFsS) on the Maze2d environment as depicted in Figure 2 of our main paper. We define $B(\pi_θ)$ in Wu et al. [1] as panel (a), referred to as the user-specified Behavior Descriptors. Panel (b), (c), and (d) demonstrate the outcomes of this setup. Panel (b) showcases the trajectory of the target agent, which closely follows the target behavior, highlighting the efficacy of our method in embedding and controlling specific agent behaviors. In contrast, Panel (c) and (d) depict the trajectories of other agents who were not given the matching bonus but were still subject to the dataset and unique behavior rewards. These agents exhibit diverse behaviors, diverging significantly from the target, thus emphasizing the diversity achievable under our framework.

We thank reviewer vcoR for the insightful comments. Below, we address the questions raised by the reviewer. We hope the replies could help the reviewer further recognize our contributions. Thank you.

Q1.I am not sure the illustrative example given in Figure 2 is the best scenario to demonstrate behavioral diversity as the demonstrated policies appear relatively similar despite attempting to showcase distinct behaviors.

A1: In Figure 2, diversity refers to the differences in behavior among policies, not variations within the same policy across different trials. Each policy is represented by a unique trajectory pattern, demonstrating how it navigates the environment. For instance, Policy 2 tends to navigate in larger loops, often closer to the upper walls of the environment. In contrast, Policy 3 exhibits behavior characterized by tighter loops and a propensity to stay closer to the lower walls. The diversity we aim to illustrate is the distinct navigational strategies adopted by different policies when placed in the same starting conditions.

Q2. The empirical evaluation's scope is limited, focusing on only three MuJoCo benchmark tasks, which may not fully validate the method's generalizability.

A2: In our offline reinforcement learning experiments, the type of training data is a crucial factor for assessing model robustness against different datasets. We utilized both the Standard D4RL dataset and the Diverse D4RL dataset across three MuJoCo tasks, creating a total of 18 distinct experimental conditions categorized into medium-expert, medium-replay, and medium settings. In this rebuttal, we extended our evaluation to the Atari domain, where our comparison is primarily with SORL, as it is the only baseline method in our study that has also been tested on Atari environments. We executed the official SORL code available on their GitHub repository to ensure a fair comparison. For this part of our study, we set $\lambda$( the weight for $U_{\text{Behavior}}$) to 1. We trained a model with three different policies and tested each across 10 trajectories using three random seeds to assess variability and consistency in performance. The results from these experiments indicate that our method not only performs well across various conditions in MuJoCo tasks but also extends effectively to other environments, including those with discrete action spaces, such as Atari.

Q3. The theoretical transition from path diversity to behavior diversity, while conceptually reasonable, requires more justification, particularly regarding equivalence conditions and underlying assumptions.

A3: The theoretical transition from path diversity to behavior diversity does not rely on any assumption. Specifically, by adapting the mutual information concept from Equation 4 to the context between the state $\mathcal{S}$ and the policy $\Pi$, we obtain $I(\mathcal{S}; \Pi) = H(\mathcal{S}) - H(\mathcal{S}|\Pi) = \mathbb{E}_{(\pi, s) \sim p(\Pi, \mathcal{S})} \left[ \log \frac{p(s|\pi)}{p(s)} \right]$. Given the non-negative nature of mutual information, this formulation supports the validity of Inequation (8).

Q4. How does dataset quality impact the method's effectiveness in promoting behavioral diversity?

A4: Thank you for raising this important question. We interpret dataset quality here as referring to the capability of the behavior policy (e.g., Medium-Expert, Medium, Medium-Replay). If our understanding is incorrect, please kindly let us know.

To better illustrate, we summarize the results from Table 1 in our paper into a concise table. Since Medium-Replay represents data collected during the training of a policy, it inherently includes samples from both strong and weak stages of the policy (or equivalently, different policies). This makes the diversity within the dataset more complex and less controllable. Therefore, for this discussion, we focus on the Medium-Expert and Medium datasets.

For our proposed method, the diversity within the dataset has a more significant impact on the resulting policy’s behavioral diversity, while the dataset's quality has minimal influence. In contrast, for baseline methods, the dataset quality plays a more significant role in determining the behavioral diversity of the policies. Fundamentally, higher data quality tends to reduce behavioral diversity in baseline methods, as can be observed in the results for DIVEOFF and CLUE. Interestingly, SORL demonstrates the opposite trend, which may be attributed to its policy performance when applied to Diverse D4RL datasets.

Q5. Could the authors provide a comprehensive hyperparameter analysis for $\lambda$ and $N$ to better understand their influence on performance?

A5: In our model, $\lambda$ serves as the weight for $U_{\text{Behavior}}$, balancing the emphasis between reward optimization and diversity. A higher value of $\lambda$ might overly prioritize diversity at the expense of achieving optimal rewards. Throughout our experiments, we set $\lambda$ to 1.0 across all offline datasets. An exception was made for the medium-expert Walker2d environment in the Diverse D4RL dataset, where it was adjusted to 0.5. Regarding the parameter $N$, which represents the number of ensembles, its primary role is to mitigate the selection of highly uncertain actions by the policy. A smaller $N$ could potentially allow for the selection of such uncertain actions, negatively impacting performance. In our experiments, we adopted the same settings used in EDAC for $N$. Specifically, $N$ is set to 10 for the HalfCheetah and Walker tasks, and 50 for the Hopper task.

Q6. How might the learned behavioral diversity translate to improved performance in downstream tasks, for example, few-shot adaptation?

A6: The learned behavioral diversity in our model holds potential advantages for improving performance in downstream tasks, such as few-shot adaptation. This capability arises from the model's ability to experience behaviors outside the dataset, providing it with a better generalization ability to new tasks.