Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning

Question 1: The intuition behind BE did not came across well. Also, why would one want a different weighting of high and low-prob. values?

Response to Question 1: As illustrated in Fig. 1, different probability weightings induce a variety of entropy valuations. When BE is used as an exploration objective, this fact will yield a variety of policies. On one end of the spectrum, when BE valuation is low an exploration policy tends to ignore low probability areas and focuses instead on refining the coverage of high probability areas, thereby creating denser and more detailed coverage. On the other end of the spectrum, where BE valuation is high an exploration policy gets attracted to low probability areas and focuses on refining the coverage of low probability areas, thus creating sparser and wider coverage. We have modified the introduction in the revision to clarify the motivation for using BE for data generation. We have also included a new visualization and additional discussion of the BE reward function from eq. (24) in the revision (see Fig. 13) to provide further insight into how BE differs from SE and how it might be expected to affect coverage.

Question 2: The graphics look nice, but in the context of beh. generation and exploration, there is no parametric underlying distribution (like Bernoulli) but essentially, one is trying to achieve uniform distribution over the state-space. An illustration on how this exploration strategy really performs differently and how would be interesting.

Response to Question 2:

We emphasize to the reviewer that the qualitative coverage results in Section 5 and the appendix (see Figs. 3, 9, 10, 11, 12) provide extensive visualizations using both $t$-SNE and PHATE plots of the coverage provided on both Walker and Quadruped by all of the data generation algorithms considered in the paper. Specifically, these visualizations illustrate the diverse kinds of coverage provided by BE for different values of $\alpha$ and how this compares with the other methods. As mentioned above, we have also included a new visualization of how our proposed BE reward (eq. (24)) varies with $\alpha$ as well as how it compares to the standard SE reward (see Fig. 13). This figure illustrates that BE produces a diverse set of rewards that can both under-weigh and over-weigh the SE reward (dotted black line), inducing a variety of exploration strategies depending on choice of $\alpha$.

Question 3: how often does the k-NN tree for fast retrieval be rebuild? After every episode?

Response to Question 3:

The $k$-NN computation is carried out on a per-batch basis. Please refer to our response to Weakness 2 for additional details.

Question 4: Since alpha<1 and q<1 work best, one could say that seldomly visited states (low probability) should give a higher weighting in the reward / entropy

Question 5: How was alpha / q selected?

Response to Questions 4 and 5:

The question of how to select the best value of $\alpha$ (for BE) or $q$ (for RE) for optimal data generation is a good one. For our experiments we have simply considered an evenly distributed range of $\alpha$ and $q$ values and, as the reviewer observed, the values $\alpha, q < 1$ lead to superior offline RL performance on downstream tasks. As discussed in our response to Question 2, some intuition about the effect of parameter choices is provided by Fig. 13 provided in the revision, but without prior information about the domain and the specific downstream tasks to be solved, it is currently unclear how $\alpha$ and $q$ might be selected more systematically. Nonetheless, we believe that with either prior information about the domain and downstream tasks or feedback during the unsupervised RL portion of training, it should be possible to adapt the values of $\alpha$ and $q$ during data generation to provide superior coverage for learning the downstream tasks at hand. This is an interesting direction for future work.

Question 6: How does the method scale to high-dimensional input?

Response to Question 6:

Our method scales well to high-dimensional state spaces due to its use of APT as its base algorithm. As discussed in our submission beginning around line 384, we used the APT algorithm as our base method for maximizing BE, RE, and SE. One of the key features of the APT method is that it learns a feature mapping into a feature space of user-defined dimension, then performs SE maximization using $k$-NN estimation over the resulting feature space. This allows the APT method to handle potentially high-dimensional state spaces via user selection of an appropriate feature space dimension and, since we use APT as our base algorithm, we enjoy this feature as well.

We thank the reviewer for devoting their time and effort to reading and evaluating our manuscript. We appreciate the comments and feedback provided. We are pleased that the reviewer appreciated our extension of BE to continuous spaces, development of $k$-NN BE estimators, and our visualizations. We have thoroughly responded to each of the reviewer's concerns and questions below.

Weakness 1: Another hyperparemeter introduced, not unsurprising that better performance can be achieved by tuning this parameter per environment

Response to Weakness 1:

We are puzzled by this statement, which seems to oversimplify our contribution and underestimate the difficulties involved in developing principled RL-based methods for exploration. Is the reviewer suggesting that investigating families of entropies beyond SE for data generation in offline RL is not worthwhile? Any clarification the reviewer can provide on this point will be much appreciated.

In any case, we respectfully reaffirm the significance of our contribution, which we again summarize here. In this work we considered a parametrized class of entropies (BE parametrized by $\alpha$), but before they could be used as practical exploration objectives we first had to extend the definition of BE to continuous spaces, develop and analyze $k$-NN estimators for BE, and derive corresponding RL-amenable reward functions. These are significant theoretical contributions. Once this had been achieved, we experimentally evaluated the effect of using BE as a data generation method for offline RL and observed that the wide range of coverages provided by BE datasets led to superior performance compared with existing methods. These results together establish the usefulness and importance of BE as a data generation method for offline RL.

Weakness 2: computationally heavy (because of k-NN that needs to be rebuild every time) and no mentioning of that

Response to Weakness 2:

We thank the reviewer for raising this important concern. We want to clarify that the $k$-NN computation is carried out on each mini-batch sampled from the replay buffer. This is in keeping with the APT and Proto-RL [Yarats et al., 2021] algorithms, and, as discussed in Section 5 starting around line 386 of our submission, we used APT to maximize behavioral entropy (BE), Renyi entropy (RE), and Shannon entropy (SE). Since mini-batch sizes were fixed at $1024$ (see Table 2 in the appendix), the computational cost of computing $k$-NNs for $k = 12$ is small.

Weakness 3: no comparison to simple baselines like RND

Response to Weakness 3:

Thanks to the reviewer for raising this valid point. To address this concern we have performed additional offline RL experiments on both RND- and SMM-generated datasets for the rebuttal and have added these to the revision (see Figs. 4, 5). As summarized in Table 1 in the revision, performance on BE datasets was superior to performance on SE, RND, and SMM datasets across all tasks. We hope that these additional results satisfy the reviewer, but we also emphasize that our original submission did compare our BE-maximizing approach with the popular SE-maximizing approach of APT (the specific algorithm we compare against is the ICM-based implementation of APT, as described in Section 5) as well as an RE-maximizing variant of this algorithm. We believe that the revised experiments strengthen the contribution of our paper and again thank the reviewer for suggesting this improvement.

Weakness 4: no improvements for strong offline methods

Response to Weakness 4:

Could the reviewer please clarify what they mean by "improvements for strong offline methods"? We emphasize that the scope of our submission is the application of BE to data generation for offline RL. We do not claim to improve existing offline RL methods or to devise a new offline RL method. This is definitely an important question, but is out of scope for the present submission.

Thanks again for your time and effort in reviewing our paper. We are following up to ensure that our response and revision have addressed all your concerns. We are happy to address any remaining questions before the end of the discussion period tomorrow.

We are following up to ensure that our response to your last round of questions has addressed all your concerns. Please let us know if there are any final questions before the end of the discussion period today.

We thank the reviewer for devoting their time and efforts to reading and evaluating our manuscript and we sincerely appreciate the comments and feedback provided. We are delighted that the reviewer found our work well-motivated, novel, and that they appreciated our figure design. We have thoroughly responded to all reviewer concerns below. Before addressing these in detail, we first provide general comments.

General comments:

We first note that in their weaknesses section the reviewer focused almost exclusively on the qualitative coverage experiments (t-SNE and PHATE visualizations) presented in the first half of our experimental results. We emphasize that our theoretical development and offline RL experiments make up a major part of our contribution (see general response to all reviewers). We feel that the reviewer may have undervalued these contributions.

Regarding our qualitative experiments, it is important to note that there is currently no consensus on what constitutes "good" coverage in the unsupervised and offline RL communities. In light of this, the main point of our qualitative coverage experiments is to illustrate the wide variety of different kinds of coverage that can be achieved using BE compared to RE, SE, and now also RND and SMM (see Fig. 12 in the revision).

In the reviewer's concerns, the reviewer appears to be thinking of coverage in distance-based, volumetric terms: that "good" coverage is achieved when points in the dataset are far apart or cover a large volume within the state space. We agree this is a potentially useful notion of coverage, so in the revision we have proposed a volumetric coverage metric and provided a quantitative comparison of coverages provided by all datasets we considered (see Fig. 8 in the revision). These new comparisons indicate there may be some positive correlation between volumetric coverage and downstream offline RL performance, but the results are inconclusive and merit future investigation beyond the scope of the present work.

Weakness 1.1: Could authors explain, in Fig3, why visualizing each experiment using separate plots is a valid metric of state coverage? ... As state trajectories go through non-linear projection, the distance between points more reflect their similarity rather their true distance, thus making it hard to say more spanned points covers large state space (I may be wrong on this). Regarding this, one way may be to visualize trajectories of some points and see if they are indeed very distinct states. Another way I would like to see is to run experiments on some 2d domains so we can visualize without PHATE.

Response to Weakness 1.1:

As mentioned above, the main point of our qualitative coverage experiments is to illustrate the wider variety of different coverages achievable using BE compared with RE, SE, RND, and SMM. We are not claiming that these qualitative results somehow provide a quantitative metric evaluating state space coverage. As discussed in our general remark, there is no agreed-upon quantitative metric for assessing "good" coverage.

Regarding the PHATE and t-SNE plots, the reviewer is correct that the non-linear projections used in the $t$-SNE and PHATE computations are not distance preserving and the plots from Fig. 3 need not scale to reflect actual distances or volumetric notions of coverage. This is natural, however, as the main purpose of these kinds of visualization techniques is to reveal trends characteristics in data that may not be apparent in quantitative methods.

Finally, to address the reviewer's concerns regarding lack of a distance- and volume-preserving quantitative coverage metric, we have proposed a volumetric coverage metric and used it to provide quantitative coverage comparisons in the revision (see Fig. 8). Our volumetric coverage notion considers the radius of the smallest hypersphere containing all points in the dataset (a well-studied problem in computational geometry, computed using Welzl's algorithm). See the discussions in our general remark above and following Fig. 8 in the revision for details.

Weakness 1.2: As each experiment is visualized separately, the PHATE may use different projections in different subfigures, making it hard to say points indeed spread out or the projection makes them look spread out. Regarding this, one more fair way is to draw PHATE plots on trajectories across all experiments to ensure they share the same projection.

Response to Weakness 1.2:

We thank the reviewer for mentioning this, as our description of the $t$-SNE and PHATE computation procedure was insufficiently clear in the original submission. When generating our visualizations we performed the $t$-SNE and PHATE plot computations for all datasets simultaneously, specifically due to the issue the reviewer has raised. This ensures that the projections used in the visualizations are the same across all datasets, providing a fair comparison across datasets. Since it was difficult to visualize the 19 different datasets on a single plot, we simply chose to visualize them individually as subplots, shown in Figs. 3, 9, 10, 11, 12 in the revision. We have added a footnote on page 8 of the revision clarifying this issue.

Weakness 2.1: I do not see singnificant difference between SE and BE for Quadruped coverage, is there any? If not, why that's the case?

Response to Weakness 2.1:

The reviewer raises a good point. Since PHATE and t-SNE are qualitative visualization methods involving learned, nonlinear projections, they may not reveal significant structure for all domains. In our case, we suspect that, since the dimension of the Walker environment (N=17) is much smaller than Quadruped (N=78), $t$-SNE and PHATE are simply able to discern patterns in the datasets better for Walker. In any case, the new quantitative coverage comparisons that we have provided for the rebuttal demonstrate that the diversity of quadruped datasets produced using BE is still higher than other methods in terms of variety of volumetric coverage values achieved (see Fig. 8 in the revision).

Weakness 2.2: In offline RL experiments, do you train each method until it converges? Authors mentions "we performed just 100K offline training steps", but I thought it's more fair to compare performance under convergence

Response to Weakness 2.2:

In Fig. 4 we report the maximum performance achieved by the offline RL algorithm over the course of 100K training steps. This is common practice, since overfitting the policy to the offline dataset is a persistent issue in offline RL, making simply reporting performance at the end of training not representative of actual performance achieved. To illustrate the sufficiency of 100K offline training steps for our results, we conducted additional ablation studies (see Fig.6 in appendix) comparing the effect of performing 100K vs. 200K offline RL training steps on performance. For these experiments, we considered 3M-element SE datasets and compared all three offline RL algorithms on all five tasks. The marginal improvements observed suggest that performing additional offline RL training has at best marginal effect on downstream task performance.

Weakness 2.3: The authors motivates in the intro that better state coverage should lead to better offline RL performance, but in Fig 6, the leading to best coverage ($\alpha=5$) typically has the worst performance across all alpha, why that's the case? Is the state coverage metric wrong or the assumption of better state coverage leading to better performance wrong or becase of other reasons?

Response to Weakness 2.3:

We again emphasize that the core message of our qualitative results is that BE provides more diverse coverage than the other methods considered. From this perspective, the difference in the coverage provided by $\alpha = 5$ compared with the coverage provided by $\alpha <= 1$ as well as that provided by SE, RE, RND, and SMM is what is important, not the downstream performance achieved for any particular parameter value. We do not claim in our original submission or revision that any particular parameter value or visualization will correspond to superior offline RL performance. We do claim that data generation methods that achieve more diverse dataset types (like the BE datasets visualized in Fig. 3) will lead to superior offline RL performance, and this hypothesis is demonstrated by the offline RL experiments presented in Fig. 4.

Suggestion 1: In intro, authors motivates the usage of BE by only saying "it's widely used in behavioral economics to model human probability perception" and the results from Suresh et al. (2024). This feels void and doesn't explain why it could helps. There is no explanation why reweighting the entropy (making it more uniform or sharper) could lead to better coverage.

Response to Suggestion 1:

We thank the reviewer for this suggestion and have modified the introduction to provide a more detailed motivation of why BE can be expected to lead to datasets with superior coverage (see the second paragraph in the introduction, highlight in blue). We have also included a visualization and additional discussion of the BE reward function from eq. (24) in the revision (see Fig. 13) to provide further insight into how BE differs from SE and how it might be expected to affect coverage.

Suggestion 2: L100, "We derive practical RL methods", it's more of a different reward/objective, as you still use an existing RL algorithm.

Response to Suggestion 2:

We thank the reviewer for this suggestion. We have modified the second bullet point in the contributions summary in the introduction accordingly (see revision).

Question 1: What is admissible generalized entropies and why "admissible" and "generalized" are important?

Response to Question 1:

We thank the reviewer for the question and will clarify briefly here (for a detailed discussion we refer the reviewer to Suresh et al. (2024) and Amig´o et al., 2018). As discussed at the beginning of Section 2, a functional $H$ is an admissable generalized entropy if it satisfies the first three Shannon-Kinchin axioms, which stipulate that $H$ is (i) continuous in its parameter, (ii) uniquely maximized by the uniform distribution, and (iii) remains unchanged when known outcomes are added to the distribution. These conditions are particular important when the generalized entropy is to be used as an exploration objective, as they ensure that the functional well-behaved as an optimization objective and seeks out uniform coverage.

Thank you very much for engaging with us during the discussion period and for raising your score. Please let us know if addressing any additional concerns could assist in improving your evaluation of our paper. We are happy to engage in further discussion if needed.

We thank the reviewer for devoting their time and effort to evaluating our submission, and we sincerely appreciate the insightful feedback provided. We are pleased that the reviewer found our contribution well-motivated, theoretically grounded, and well-presented. We have thoroughly responded to each of the reviewer's concerns below.

Weakness 1: The authors propose a KNN-based estimator with convergence guarantee. I wonder in practice how $k$ is chosen and if KNN introduces huge computation overhead compared to other baselines. For instance, if SE and RE have the same computation budget as BE, will they collect more data and hence perform better?

Response to Weakness 1:

We thank the reviewer for raising these important questions. Regarding the choice of $k$, in our experiments we chose $k = 12$ (see Table 2 in the appendix) as used in the ExORL framework [Yarats et al., 2022] to ensure that our experiments remained comparable with benchmark APT implementation considered in that framework. In general, however, the choice of $k$ is a matter of trial and error: early work on $k$-NN estimators for Shannon entropy experimentally found $k = 4$ to lead to reasonable performance [Singh et al., 2003], while the paper proposing APT [Liu and Abbeel, 2021] used values $k = 3, 5,$ and $10$ depending on the environment. From what we know of the literature on $k$-NN entropy estimators, relatively small values $k \leq 15$ are most common, even in high-dimensional spaces.

Concerning the computational overhead from using $k$-NN estimation, it is important to note that the $k$-NN computation is carried out on each mini-batch sampled from the replay buffer. This is in keeping with the APT and Proto-RL [Yarats et al., 2021] algorithms, and, as discussed in Section 5 starting around line 386 of our submission, we used APT to maximize behavioral entropy (BE), Renyi entropy (RE), and Shannon entropy (SE). Since mini-batch sizes were fixed at $1024$ (see Table 2 in the appendix), the computational cost of computing $k$-NNs for $k = 12$ is small. We emphasize that SE and RE were estimated using the same underlying $k$-NN computation as that used in BE, ensuring a fair comparison between all three methods. In particular, this means that all three methods compared (SE, RE, BE) had the same computation budget.

Weakness 2: The experiments only generate 500K elements for each task, which might be not enough for the offline RL to learn a good policy. Can we test if the performance of BE, SE, and RE will scale with more data?

Response to Weakness 2:

The reviewer is right that we only generated 500K-element datasets for each combination of exploration algorithm and environment, which contrasts sharply with the large (10M+ element) datasets often considered in offline RL settings. However, as described in Section 5, lines 385-391, despite using only 500K-element datasets we achieved comparable performance on downstream tasks to benchmark algorithms trained on 10M-element datasets in the ExORL framework. Importantly, the superior downstream performance on BE datasets compared with SE and RE datasets that we observed indicates that BE can be used to achieve better data- and sample-efficiency when used to perform dataset generation for offline RL (see lines 389-391). This fact illustrates a key strength of our approach. To highlight this feature of our approach while simultaneously addressing the concerns of the reviewer, for the rebuttal we have performed additional offline RL experiments on a 3M-element SE-generated dataset and have revised the submission accordingly (see Figs. 4, 5). These experiments demonstrate that 500K-element datasets generated using BE continue to lead to superior downstream offline RL performance than 3M-element datasets generated using SE.

Question 1: While this paper focuses on the offline RL setting, I am curious about whether this entropy-based exploration can be applied in the online RL setting and how it performs compared with other online exploration methods.

Response to Question 1:

We agree with the reviewer that BE-based exploration for the online RL setting is an interesting direction worth further investigation. Given our focus on data generation for offline RL, however, we believe this important question is out of scope for the present submission and best left for future work.

Thanks again for your time and effort in reviewing our paper. We are following up to ensure that our response and revision have addressed all your concerns. We are happy to address any remaining questions before the end of the discussion period tomorrow.

Thank you very much for engaging with us during the discussion period. Please let us know if addressing any additional concerns could assist in improving your evaluation of our paper. We are happy to engage in further discussion before the end of the discussion period tomorrow, if needed.

Question 3: Along with the qualitative visualizations you have provided (e.g PHATE/tsne), could a quantitative analysis of the dataset state/action coverage be done to compare the different dataset generation approaches?

Response to Question 3:

The reviewer raises an important open problem in the unsupervised RL and offline RL literatures: there is currently no consensus in these communities on the best way to quantitatively evaluate how "good" an exploration objective is or how "good" the coverage of a given dataset is. One symptom of this is the proliferation of different exploration objectives in unsupervised RL: if there were agreement on the best metrics for quantifying the goodness of exploration (e.g., Shannon entropy), then exploration methods would naturally focus on optimizing those metrics; this is not the case, however, highlighting the aforementioned lack of consensus in the unsupervised RL community. In the offline RL setting, many offline RL datasets exist, but the methods used to generate the datasets vary widely. In the widely used D4RL benchmark [Fu et al., 2021, https://arxiv.org/pdf/2004.07219], for example, due to lack of an accepted metric for evaluating dataset coverage, a hodgepodge of random, partially trained, and expert policies are used for dataset generation (see the appendix of [Fu et al., 2021] for a description of the domains and dataset generation methods used). This lack of principled methods for dataset generation for offline RL is one of the primary motivations underlying the development of the ExORL framework.

Despite this question remaining an open problem, to more fully address the reviewer's concerns in our rebuttal we have proposed a quantitative metric for evaluating dataset coverage and used this metric to provide a quantitative comparison of the datasets we considered. The quantitative metric we propose is the volume of the smallest hypersphere covering the dataset, which we compute using Welzl's algorithm, a standard procedure from the computational geometry literature for solving the smallest-circle problem. We have included these quantitative comparison and a discussion of them in Figure 8 in the appendix of our revision.

Weakness 4: Additionally, there are a myriad of other exploration objectives people consider such as RND, Count Based Exploration, etc. It would be good to empirically compare against these approaches for the synthetic dataset generation.

Response to Weakness 4:

Thanks to the reviewer for raising this valid point. Importantly, it is also computationally feasible, since adding a single additional data generation method to our original experiment setup (c.f., Figure 4) only results in an additional 75 trials. We have therefore performed additional offline RL experiments on both RND- and SMM-generated datasets as well as a 3M-element SE-generated dataset for the rebuttal and have provided an updated revision presenting these results (see Figures 4 and 5 in the revision). We hope that these additional results satisfy the reviewer, but we also emphasize that our original submission did compare our BE-maximizing approach with the popular SE-maximizing approach of APT (the specific algorithm we compare against is the ICM-based implementation of APT, as described in Section 5) as well as an RE-maximizing variant of this algorithm. We believe that the revised experiments strengthen the contribution of our paper and again thank the reviewer for suggesting this improvement.

Question 1: For BE, is there a clear selection procedure for $q$ empirically that would transfer across algorithms/tasks?

Response to Question 1:

The question of how to select the best value of $\alpha$ (for BE) or $q$ (for RE) for optimal data generation is a good one. For our experiments we have simply considered an evenly distributed range of $\alpha$ and $q$ values and we have observed empirically that values $\alpha, q < 1$ lead to superior offline RL performance on downstream tasks. Without prior information about the domain and the specific downstream tasks to be solved, it is currently unclear how $\alpha$ and $q$ might be selected more systematically. However, we believe that with either prior information about the domain and downstream tasks or feedback during the unsupervised RL portion of training, it should be possible to adapt the values of $\alpha$ and $q$ during data generation to provide superior coverage for learning the downstream tasks at hand. This is an interesting direction for future work.

Question 2: Currently, the approach relies on a k-NN estimator, which has limitations as the state dimensionality increases or the size of the dataset is large. As offline RL scales to be used with more realistic domains, these concerns will be more common place. Are there tradeoffs with using this as an estimator?

Response to Question 2:

There are two main reasons why the use of $k$-NN estimation remains computationally tractable even for large datasets and high-dimensional state spaces. Concerning large datasets, the $k$-NN computation is carried out on each mini-batch sampled from the replay buffer. This is in keeping with the APT [Liu & Abbeel, 2021] and Proto-RL [Yarats et al., 2021] algorithms, and, as discussed in Section 5 starting around line 386 of our original submission, we used APT to maximize BE (as well as RE and SE). Since mini-batch sizes were fixed at $1024$ (see Table 2 in the appendix), the computational cost of computing $k$-NNs for $k = 12$ (this $k$ was chosen to remain consistent with the ExORL framework) is fairly small. Regarding high-dimensional state spaces, one of the key features of the APT method is that it learns a feature mapping into a feature space of user-defined dimension, then performs $k$-NN estimation over the resulting feature space. This allows the APT method to handle potentially high-dimensional state spaces via user selection of an appropriate feature space dimension and, since we use APT as our base algorithm, we enjoy this feature as well.

Weakness 1: Some of the empirical results aren’t compelling such as the $\alpha/q$ sweep done in Figure 4, where the gap from BE and Shannon/Renyi Entropy is minimal or within the margin of error for domains, which questions the advantages of the dataset generated.

Response to Weakness 1:

The core message of our offline RL results is that BE datasets provide at best superior and at worst competitive offline RL performance on downstream tasks when compared with alternative dataset generation methods, and that BE is therefore an important new exploration objective to consider when performing data generation for offline RL. We respectfully disagree with the reviewer that the experiments presented in Figures 4 (and 5) are not compelling, especially in light of the additional experiments performed for the revision. As summarized in Table 1 and discussed in Section 5, BE datasets lead to superior performance over SE, RND, and SMM datasets across all tasks, while BE datasets lead to superior performance over RE datasets on 4 out of 5 tasks and remain competitive on the fifth task. As summarized on lines 511-514 of our submission, in our experiments

"best offline RL performance on BE-generated datasets clearly exceeds best performance on RE datasets on 13 out of 15 task-algorithm combinations and best performance on SE, RND, and SMM datasets on 15 out of 15 task-algorithm combinations."

Given that offline RL training for each task-algorithm combination was evaluated over 5 independent replications, we emphasize that the superior offline RL performance observed on BE-generated datasets is statistically meaningful and that the usefulness of BE as an exploration objective for dataset generation is empirically well-supported by our experiments. In addition, in our revision we have shown that this trend continues to hold for additional dataset generation methods (see Figures 4 and 5 in the revision).

Weakness 2: Additionally, it would be good to check the consistency of the dataset generation, where the dataset generation process is repeated for each $\alpha$ or $q$, $N$ times and performance is aggregated across these generation attempts.

Response to Weakness 2:

While we agree with the reviewer that verifying the consistency of the data generation process in this way would be ideal, we emphasize that the computational burden that this entails for statistically meaningful values of $N$ (e.g., $N \geq 5)$ is prohibitive. This limitation is especially acute in our case, as the number of data generation methods we considered in the original submission is 14 (each $\alpha$ and $q$ value corresponds to a distinct dataset generation method) which is large compared to previous work: only 8 dataset generation methods were considered in both the URLB [Laskin et al., 2021] and ExORL [Yarats et al., 2021] benchmarks. The reasons for this are outlined in the Insufficient experiments issue discussion above. We furthermore highlight that the $N=1$ approach to dataset generation that we have followed is standard in the unsupervised RL literature due to precisely the same practical limitations that we have described (see, e.g., the dataset generation procedures used in URLB and ExORL).

Weakness 3: Additionally, the domains evaluated are state based and a limited subset of the Mujoco benchmark. It would be good to see the clear benefits of BE-generated datasets across multiple tasks and algorithms.

Response to Weakness 3:

We again agree with the reviewer that considering additional environments would be ideal. Nonetheless, we reiterate that the computational burden this entails is prohibitive for the present work. Specifically, notice that for each additional environment that we consider we need to generate new datasets for each data generation method, then train multiple independent replications of each offline RL algorithms on each downstream task on that environment. Assuming 14 dataset generation methods, 3 downstream tasks, 3 offline RL algorithms, and 5 offline RL seeds, this entails $675 = 15 \times 3 \times 3 \times 5$ additional trials for each new environment considered.

We thank the reviewer for devoting their time and effort to evaluating our submission, and we sincerely appreciate the insightful feedback provided. We are pleased that the reviewer recognized the novelty of our work and appreciated the insights provided by our experimental results. We have thoroughly responded to each of the reviewer's concerns below.

Insufficient experiments issue:

Before addressing each of the reviewer's concerns in detail, we first highlight that one of the reviewer's primary concerns is that insufficient experiments were conducted (three out of four weaknesses described by the reviewer relate to this). The reviewer proposes several ways to expand the experiments to remedy this, including: (i) replication of dataset generation, (ii) additional data generation algorithms (RND, etc.), (iii) additional environments, (iv) additional offline RL algorithms. While we agree that expanding the experiments to include all these desiderata is ideal and would make the experimental results extremely robust, we emphasize that the computational expense this would incur is extreme. We note that we have performed additional experiments and significantly revised our submission to address issue (ii), as detailed in the general response to all reviewers, but we argue below that requiring (i), (iii), and (iv) as well is computationally infeasible.

To provide the reviewer with additional insight into the computational effort involved, we present details in table form below. First, consider the following table adding up the total number of individual trials required to perform the offline RL training portion of the experiments (illustrated in Fig. 6 in the appendix) in our original submission.

Each of the 1050 (= 420 + 630) trials required a training time of approximately 2 GPU hours on our hardware. For the dataset generation portion of our experiments, we trained 28 (= data generation methods $\times$ environments) separate data generation policies, each of which required 8 GPU hours. In total, approximately 2324 GPU training hours were required for our experiments.

To achieve desiderata (i)-(iv) above, suppose that we: (i) repeat data generation $N = 5$ times for each data generation method; (ii) include RND [Burda et al., 2019], SMM [Lee et al., 2019], and DIAYN [Eysenbach et al., 2019]; (iii) consider 2 additional environments with 3 downstream tasks apiece; (iv) evaluate 2 additional offline RL algorithms. The resulting number of offline RL trials is given in the following table.

This gives a total of 23375 (= 4250 + 19125) distinct trials. Including the 340 (= data generation methods $\times$ data generation replications $\times$ total number of environments) data generation policies that need to be trained and assuming compute times are similar to the experiments we ran for the original submission, this would entail 49470 GPU hours of training altogether. We believe that it is unrealistic to expect this level of experimental cost even from a primarily experimentally focused submission. Given that core parts of our overall contribution are theoretical and algorithmic (see Sections 2-4), we maintain that this is especially true in our case.

Thanks again for your time and effort in reviewing our paper. We are following up to ensure that our detailed response and revision have addressed all your concerns. We are happy to address any remaining questions before the end of the discussion period tomorrow.

Thank you very much for engaging with us during the discussion period and for raising your score. Please let us know if addressing any additional concerns could assist in improving your evaluation of our paper. We are happy to engage in further discussion if needed.