We sincerely thank R-mB11 for their constructive feedback and for recognizing several key strengths of our work. We are pleased that you acknowledged our AntMaze-Extreme dataset, which underscores the robustness of our evaluation framework in longer-range navigation settings. We appreciate your recognition of how our work builds upon HIQL by incorporating tokenization methods and a planner policy that effectively breaks down the problem into k distinct navigation tasks, leading to SOTA results in long-range navigation. Lastly, we are thankful for your positive remarks regarding the comprehensive documentation of our code.

1. For the description of the training and inference procedure

W1. The paper lacks a methodology diagram describing the entire training and inference procedure, which makes the paper difficult to follow.

For the inference diagram, we kindly refer R-mB11 to Figure 3 which describes the inference pipeline of our model. Additionally, we understand R-mB11 concerns about the lack of diagrams concerning the training procedure. As all the components of QPHIL are trained sequentially, we updated subsection 4.1 with a statement to better inform of such design.

2. On the Selection of Evaluation Benchmarks

W2. The experiments lack other navigation environments and results are only shown on Antmaze.

We appreciate Reviewer R-mB11 for highlighting this concern. Since multiple reviewers have raised this issue, we kindly direct you to the "Answer to All Reviewers" section, specifically the "On the selection of evaluation benchmarks" subsection, where we address this topic in detail.

3. About the need of two distinct low policies

W3. For the low level policy, and are distinct policies which are both generating low level actions to reach to and respectively. The need for having two different policies is unclear and sparsely mentioned in the text.

Q2. Could you elaborate on W3 and explain the reason behind having two distinct low level policies. Could you show some ablations which provide empirical reasons for having.

Our method can be understood as decoupling the long-range goal reaching task into a long-range landmark reaching task and a short-range goal reaching task. Discretizing the state space allows for easier navigation across large environments but is only meant to bring the agent to the landmark of the goal. Hence, the discrete-goal conditioned policies (planner and low level) are high performing for long range planning and navigation but are not meant to perform precise goal reaching. The continuous-goal conditioned policies, on the other hand, struggle to perform long term planning but are precise enough to reach specific near goals. Hence, by first solving the long range navigation with the discrete-goal conditioned policies and then finishing by reaching precisely the goal, we can increase overhaul navigation performance. Additionally, using a single low-level policy requires consideration of a double-conditioning featuring the absolute goal position and the next landmark to reach. We believe this approach to be bound to underperform as absolute goals will bring noise to landmark reaching.

4. About Figure 4

W4. In the tokenizer, the contrastive loss penalizes temporal closeness of the states in the latent space. W4.1. The tokenization example in Figure 4, has tokens broken into very non temporally close spaces as token 47 and 46 reach areas near token 43, and token 12 reaches near 16 and 17. W4.2. It’s also unclear how this loss “aligns with the walls thanks to the contrastive loss.” as mentioned in the caption of Figure 4.

Q1. In Figure 4, please explain the tokenization in more detail wrt. the W4. Also could you please elaborate on both the parts of W4.

We thank the reviewer for pointing this out. We updated the Figure 4 to a better tokenization. We also refer R-mB11 to the experiments of appendix D.4 which assesses the impact of the different quantizer losses on the resulting tokenization.

We greatly appreciate R-E6v9 for their encouraging feedback and for recognizing the strengths of our work. We are delighted that you found our paper well-motivated and clearly presented, and that our experimental results effectively support our claims of enhanced performance in longer-horizon tasks. Furthermore, we are thankful that our discretization scheme and explicit trajectory stitching for high-level planning from offline data were viewed as innovative approaches with broad applicability to various settings.

1. About codebook size and continuous planning

W.1 The paper is missing some details on the impacts of codebook size and tuning of the contrastive loss weights on the performance of the approach, these parameters seem integral to the contribution of the paper and also for broader applicability. As identified by the paper the subgoal step parameter of k is important in balancing the signal-to-noise ratio in low-level and high-level updates, as a consequence of replacing the training of high level policy with BC by a transformer it might have been beneficial to have very granular codebooks?

We thank the reviewer for suggesting such experiments. As this interrogation is shared with R-XZDy, we kindly refer to the "Answer to all reviewers" part, in the "About codebook size and continuous planning" section which deals with the impact of the codebook granularity on QPHIL’s overhaul performance.

2. About replanning

Q.1 Are all the results of QPHIL presented operating on open loop high level plans synthesized by the transformer policy on seeing just the first state?

We thank the reviewer R-E6v9 for pointing out this need for clarification. As this question is also raised by R-XZDy, please refer to the "Answer to All Reviewers" section, particularly the "About replanning" subsection, where we explore the effect of replanning integration on QPHIL’s performance.

3. On the impact of contrastive loss

Q.2 The hyperparameter for VQ-VAE suggests a very high coefficient for contrastive loss over reconstruction loss – what are suitable scales to balancing these different losses? Did they have to be tuned for different scales of the maze?

As you mentioned the coefficients for the learning of the quantizer (the commit loss coefficient of the VQ-VAE, the contrastive loss coefficient as well as the reconstruction loss coefficient) need tuning. As we use the same coefficients for each loss for each maze shape, we can argue that our set of values are generalizable to every shape and size of the tested mazes. The tuning of those losses was the result of hyperparameter search. Moreover, following R-E6v9 suggestion, we added in appendix D.4 a more comprehensive analysis of the impact of the contribution of each loss to the learning of the space discretization.

We are thankful to R-XZDy for their review. We are thankful that our work is acknowledged for its significant improvement over the previous methods on difficult long range navigation tasks, thanks to our choice of a contrastive loss function that allows an adaptive discretization of the state space, which can be leveraged for planning thanks to data augmentation through stitching.

1. On the selection of evaluation benchmarks

W.1 A primary concern with the results is that they are limited to the navigation domain, specifically the AntMaze environment. This raises questions about the generalizability of the method to other settings, such as the Kitchen or Calvin environments evaluated in the HIQL paper. Expanding the evaluation to include diverse tasks would provide a clearer understanding of the method's broader applicability.

We appreciate R-XZDy for bringing up this concern. As this question has been raised by multiple reviewers, we kindly direct you to the "Answer to All Reviewers" section, particularly the "On the Selection of evaluation benchmarks" subsection, where we address this topic in detail.

2. About additional related works

W.2 The concept of discrete planning over learned landmarks, subgoals, or skills has been well-explored in both online RL (e.g., Choreographer [1], Dr. Strategy [2], CQM [3]) and offline RL (e.g., PTGM [4], SAQ [5], SkillDiffuser [6], TAP [7]). While I acknowledge that none of these prior works directly address the goal-conditioned offline RL setting and that they are slightly methodologically different, the novelty here appears limited and incremental.

We sincerely thank R-XZDy for their insightful feedback and for providing precise references to support their concerns regarding the novelty of our work. We appreciate the opportunity to clarify our contributions and address these points in detail. As we consider that this question might be of interest for all reviewers, we address it in the "Answer to All Reviewers" section in the "About additional related works".

3. About codebook size and continuous planning

W.3 I would suggest modifying their approach by training a VAE instead of a VQ-VAE to enable such a comparison as part of an ablation study. Alternatively, they could compare their method to a variant using a larger number of landmarks to assess how this affects performance in handling the signal-to-noise problem.

Q.1 Could you provide more details on how you selected the number of landmarks? Additionally, how does varying the number of landmarks impact performance? It would be helpful to understand how different quantities affect the system’s effectiveness.

We appreciate R-XZDy for recommending ablation experiments to compare continuous and discrete state planning. Since R-E6v9 has also raised this point, please refer to the "Answer to All Reviewers" section, particularly the "About codebook size and continuous planning" subsection, where we provide a comprehensive response.

4. About “w/ repr.” and “w/o repr.”

Q.2 In the experiments, could you clarify the terms “w/ repr.” and “w/o repr.”? I couldn’t find an explanation in the text.

We thank the reviewer for pointing out this omission. “w/ repr.” and “w/o repr.” refers to the variants of the HIQL algorithm described in [1]. HIQL “w/o repr.” uses two policies, a high level policy $\pi_{\theta_h}^h(s_{t+k}|s_t,g)$ that generates a subgoal to reach given the current state $s_t$ and the goal $g$ and $\pi_{\theta_l}^l(a_t|s_t,s_{t+k})$ that generates an action to take to reach this subgoal given $s_t$ and $s_{t+k}$. Because HIQL may handle complex high-dimensional observation spaces, generating such subgoals in raw format might be challenging. Hence, they propose to learn a representation $\phi(s)$ of the states, leading to a high level policy $\pi_{\theta_h}^h(\phi(s_{t+k})|s_t,g)$ and the low level policy $\pi_{\theta_l}^l(a_t|s_t,\phi(s_{t+k}))$, leading to the “w/ repr.” variant. We display the two in our tables for a better comparison. This is now clearly stated in the paper below Table 1.

[1] HIQL: Offline Goal-Conditioned RL with Latent States as Actions

5. About replanning

Q.3 From Figure 3, it appears that planning occurs only once. How does the system handle situations where the agent deviates from the plan? Is there any mechanism for re-planning if the agent is unable to follow the initial trajectory?

We thank the reviewer for pointing this out and kindly refer to the “Answer to all reviewers” part, in the section “About replanning”, which delves into the impact of replaning integrations to our method.

We thank Reviewer R-hhSM for their insightful feedback and for acknowledging the strengths of our approach. We are pleased that you found the idea of quantifying states into coarse landmarks for high-level planning in hierarchical reinforcement learning (RL) to be both interesting and intuitive. Additionally, we appreciate your recognition of our algorithm's significant performance improvements on the AntMaze benchmark. Your comments provide valuable validation of our methodology, and we have taken them into consideration to further refine and enhance our work.

1. On the selection of evaluation benchmarks

W.1 Although multiple variants of the AntMaze environment were considered, the algorithm was only tested in the AntMaze environment. It should be tested for additional and possibly more complex navigation tasks (e.g., visual navigation, autonomous driving). One crucial factor is to show that the proposed tokenization method can scale to high-dimensional observation space (e.g., images).

We thank R-hhSM for raising this concern. Since other reviewers have also highlighted this issue, we kindly refer you to the "Answer to All Reviewers" section, specifically the "On the selection of evaluation benchmarks" subsection, where we provide a comprehensive response.

2. On the value of the learning of the latent space

W.2 I also wonder if learning a latent space and quantizing it in the tested AntMaze environment is necessary. According to the visualized examples, the learned landmarks are mostly uniformly distributed across the maze and similar in size. Can one simply discrete the maze into lattices without learning the latent space? It also links back to the first point. The experiments would be more convincing if additional navigation tasks with high-dimensional visual observations could be included.

In the tested Antmaze environment, it is indeed possible to discretize uniformly the maze to perform planning. However, this approach would be limited because of several points. First, it requires the user to have previous knowledge about the shape of the maze, to choose small enough lattices to avoid landmarks to span across obstacles, which is not necessarily the case and is a strong hypothesis that our method doesn't require. Also, the size of those lattices could become really small for some environments, leading to an unnecessary increase of the number of tokens, resulting in more difficult planning. Additionally, we ran experiments with uniform tokenization on the antmaze-ultra-diverse-v0 dataset with a comparable amount of tokens than what we used for the main experiments. This resulted in a score of 53.5 ± 12.7, which is 10 to 20 pourcent lower than with the quantizer method. We reported the figure of the resulting tokenization as well as our explanation in appendix D.7.

3. QPHIL performance with and without data augmentation

Q.1 In some of the tasks in Table 1, introducing data augmentation hindered QPHIL's performance by a large margin. Could the authors explain why data augmentation was not effective in these cases? It makes sense to me that data augmentation does not necessarily improve performance. Still, it is counterintuitive that the data augmentation step could cause a significant drop in performance in this context.

We thank the reviewer R-hhSM for pointing out this need for clarification. As this question is also raised by R-mB11, please refer to the "Answer to All Reviewers" section, particularly the "QPHIL performance with and without data augmentation" subsection, where explain the reason behind such differences in performances as well as proposing an additional experiment to illustrate our statement.

The authors didn't provide rebuttals. So I suggest rejecting this paper.

It provides a first rigorous and diverse testing ground for QPHIL’s navigation scope, as well as being challenging even for state-of-the-art methods like HIQL.

I agree with you to some extent, but I think you still need to show how this method can perform in different environments, not just navigation similar to HIQL, they showed a comparison in the Kitchen, Calvin and Procgen environments. I believe that such results will make your results much stronger and complete.

About codebook size

The provided results look useful for understanding the required codebook size and how your method works with more codes. Thank you for providing this to answer my question.

About the continuous planning

I do not understand why you compared: BC+IQL w/ repr. vs. BC+IQL w/o repr. vs. QPHIL w/ aug. vs. QPHIL w/o aug. Are not the augmentation and representation different things? Should not you fix the augmentation and compare for example,

(with augmentation) BC+IQL w/ repr. vs. BC+IQL w/o repr. vs. QPHIL w/ repr. vs. QPHIL w/o repr

and maybe another table for

(w/o augmentation) BC+IQL w/ repr. vs. BC+IQL w/o repr. vs. QPHIL w/ repr. vs. QPHIL w/o repr?

Is not the augmentation can be applied to BC+IQL as well?

4. About performances on the smaller scale environments (2/2)

R-mB11: "Why does QPHIL not perform above SOTA in smaller Antmaze datasets? Could changing the way of tokenization help with this?"

The conducted experiments such as in section 5.2 as well as appendix D.2 show that QPHIL benefits from the discretization of the state space, allowing to break the problem into n distinct navigation problems as R-mB11 mentioned. We argue that those improvements come from the improvement of the signal-to-noise ratio of the learned value function, which worsens the quality of the high-level planning in methods such as HIQL which aims to learn a unique value function across the entire state space.

By design, HIQL imposes a tradeoff regarding the hyperparameter controlling the distance of high-level subgoals. If this distance is high, high-level value estimates will be simpler to perform (high signal-to-noise ratio) but the low-level policy will be facing harder navigation challenges (subgoals will be distant), and vice-versa.

Hence, as QPHIL learns the value function only for the low-level policy to reach the next planned landmark from its current landmark, the signal-to-noise ratio remains quite low as the long-range planning is performed with an independent planner in a discretized space. As such, for small mazes, HIQL does not suffer too much from a bad signal-to-noise ratio and consequently the two approaches display similar performances. However, for long range scenarios, HIQL’s signal to noise ratio worsens while QPHIL’s remains good, which explains why it is scalable to very long range scenarios in contrast to HIQL.

4. About performances on the smaller scale environments (1/2)

We thank reviewers R-mPVU and R-mB11 for raising the point that QPHIL does not outperform the SOTA in the smaller environments.

R-mPVU: "In many AntMaze settings, the proposed approach does not consistently outperform existing state-of-the-art methods, such as HIQL, thus limiting the evidence for its effectiveness over current approaches."

While we agree that QPHIL does not always statistically outperform previous methods on the smaller scale maps, it always remains competitive. On small environments (medium to large mazes), QPHIL obtains comparable results than HIQL and none of the approaches appears to consistently outperform the other in the results provided in the following table and in section 5.2.

Indeed, in the medium-maze, both methods display similar performances, with a slight advantage to QPHIL, while HIQL performs better on large-diverse slightly better than QPHIL, but also better than in the medium-maze setting. However, on ultra and extreme mazes, QPHIL clearly overperforms HIQL and the other methods. As such, while staying competitive in short-distance settings, the proposed approach of tokenization appears strongly beneficial for longer-range scenarios which are likely closer than possible real world applications. We also provide the corresponding performance curves in the following website: https://sites.google.com/view/qphil/home.

Alternatively, we conducted a Welch-Student test to check the hypothesis that “HIQL w/ repr.”’s mean is less than “QPHIL w/ aug.”’s one. We display below the results:

The statistical analysis, using one-tailed t-tests to determine if HIQL's performance is less than QPHIL's, provides compelling evidence for QPHIL's superior scaling in larger maze environments. Namely:

• Medium-Sized Environments: In medium-sized environments, both tests yield p-values below the 0.05 significance level (0.0444 and 0.0237), supporting the hypothesis that QPHIL outperforms HIQL.
• Large-Sized Environments: The results in large-sized environments do not support the hypothesis. Neither test achieves statistical significance, with p-values of 0.8558 and 0.9981, indicating that HIQL may perform comparably or better in these cases.
• Ultra-Sized Environments: In ultra-sized environments, one test achieves strong statistical significance (p = 0.0008), while the other remains insignificant (p = 0.0938). These results indicate QPHIL's emerging advantage in handling the increased complexity of these environments, despite some variability.
• Extreme Environments: The strongest evidence for QPHIL's advantage appears in the largest "extreme" environments. Both tests achieve extremely high statistical significance (p = 0.0000 and p = 0.0003), conclusively rejecting the null hypothesis. These results confirm QPHIL's superior performance in handling the challenges of extreme maze environments.

In conclusion, The p-value analysis clearly demonstrates QPHIL's scaling advantage over HIQL, especially in ultra and extreme environments. While the differences are less significant in smaller environments, the statistical significance of the results in ultra and extreme settings supports the conclusion that QPHIL excels in handling increasingly complex tasks.

3. About replanning

We thank reviewers R-XZDy and R-E6v9 for voicing their interrogation about the open-loop aspect of our planning. In our main evaluations, the planning occurs only once. The policy is consequently achieving subgoal landmarks sequentially, following the initially generated plan. Aware of this, we also tested a replaning strategy which consists in replaning from the current landmark to the goal when an out-of-path landmark is detected. This replanning is done by sampling several paths and choosing the shortest one. We tested this approach in the antmaze-extreme environment and we displayed the impact of this replaning in appendix D.3 though the following table:

While we observed no significant improvement in the tested maze from such replanning in the extreme play setting, our method supports replanning easily which emphasizes the great potential of the approach.

2. About codebook size and continuous planning

We thank R-XZDy and R-E6v9 for suggesting an analysis of the granularity of the codebook, first of all by comparing the differences between the continuous (or infinite subgoal number) and discrete subgoal paradigms, but also by analyzing the impact of different codebook sizes.

First, we followed R-XZDy's suggestion on analyzing the impact of shifting from discrete to continuous space planning. As our architecture corresponds to a hierarchical planning involving high level subgoal planning through imitation learning and low level subgoal reaching in a reinforcement learning framework, a simple approach for comparing both continuous and discrete subgoals would be to consider an ablated HIQL which uses behavioral cloning rather than offline reinforcement learning to learn a high-level policy. We call this approach BC+IQL and tested its performance on the antmaze-ultra datasets. In the corresponding table below, we observe similar results to HIQL, while QPHIL demonstrates better performance, highlighting the benefits of using discrete planning for longer range navigation scenarios.

We reported those new results in appendix D.2 of the revised version of the pdf. We precise that the “w/ repr.” and “w/o repr.” correspond to the addition of representation learning, while “w/ aug.” and “w/o aug” correspond to the presence of data augmentation by trajectory stitching. Additionally, as suggested by the reviewers R-XZDy and R-E6v9, we launched a new experiment to assess the impact of the codebook size on the overhaul performance on antmaze-ultra-diverse-v0, obtaining the following table, also in appendix D.2:

The first column corresponds to the number of available codebook vectors of the VQ-VAE quantizer. The second column corresponds to the number of used codebook vectors. We consider that a codebook vector is used if there exists a state from training trajectories that projects its encoding on it and that a landmark is formed by the set of states whose encodings are projected on the same codebook vector. We observe that after a given threshold on the number of available codebook vectors, their use percentage decreases, i.e. the quantizer does not benefit from additional codebook vectors to achieve a good quantization. The performance of the method appears to saturate after the threshold of 48 vectors. Regarding the antmaze-ultra map, this codebook size corresponds to a good trade-off ensuring an accurate “signal-to-noise'' ratio while stabilizing high-level commands.

6. About additional related works (2/2)

5. SAQ [5] (Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning) (already in the related work)
   • Overview: SAQ uses a VQ-VAE to discretize the action space but does not incorporate planning mechanisms.
   • Comparison: While SAQ focuses on action space discretization, our work extends this concept by applying discretization to the state space and integrating comprehensive planning mechanisms such as trajectory stitching and data-coherent path prediction. These additions enable our framework to perform effective hierarchical planning, which is not addressed by SAQ, thereby providing a more holistic solution for goal-conditioned offline RL.
6. SkillDiffuser [6]
   • Overview: SkillDiffuser utilizes diffusion-based task execution with skill abstractions for hierarchical planning, primarily in online RL settings.
   • Comparison: SkillDiffuser’s diffusion-based approach and skill abstractions differ significantly from our method. Our work introduces explicit trajectory stitching and contrastive loss, which are specifically designed to enhance planning robustness and data alignment in offline RL. These features allow our framework to better handle the unique challenges of offline environments, such as data scarcity and the need for reliable trajectory generation based on existing data.
7. TAP [7] (Transformer-Based Planning) (already in the related work)
   • Overview: TAP integrates a VQ-VAE between the encoder and decoder of a transformer, applying the transformer to sequences of (state, action, reward) tuples and planning based on expected returns.
   • Comparison: While TAP leverages transformers for planning, our approach utilizes a sequence generator tailored for offline RL that emphasizes data-aligned path prediction. Unlike TAP’s focus on expected returns, our method prioritizes planning paths that are supported by the dataset, ensuring higher success rates by avoiding low-data regions. Additionally, our incorporation of contrastive loss enhances the temporal coherence of the learned representations, further distinguishing our approach from TAP’s methodology.

In summary, while prior research has laid the groundwork for discrete planning and hierarchical RL, our work extends these concepts by introducing novel mechanisms tailored to the goal-conditioned offline RL setting. Through the integration of contrastive loss, explicit trajectory stitching, and data-coherent planning, we provide a robust and effective framework that addresses the specific challenges inherent to offline environments. These contributions not only differentiate our approach from existing methods but also advance the field by offering practical solutions for complex, long-horizon navigation tasks in offline RL.

We thank again R-XZDy for those literature suggestions and hope this detailed comparison clarifies the novel and impactful aspects of our work, demonstrating its significant advancement over existing methodologies in the field of goal-conditioned offline reinforcement learning.

[1] Mazzaglia, Pietro, Tim Verbelen, and Bart Dhoedt. "Choreographer: learning and adapting skills in imagination." ICLR, the 11th International Conference on Learning Representations. 2023.

[2] Hamed, Hany, et al. "Dr. Strategy: Model-Based Generalist Agents with Strategic Dreaming." Forty-first International Conference on Machine Learning.

[3] Lee, Seungjae, et al. "Cqm: Curriculum reinforcement learning with a quantized world model." Advances in Neural Information Processing Systems 36 (2023): 78824-78845.

[4] Yuan, Haoqi, et al. "Pre-training goal-based models for sample-efficient reinforcement learning." The Twelfth International Conference on Learning Representations. 2024.

[5] Luo, Jianlan, et al. "Action-quantized offline reinforcement learning for robotic skill learning." Conference on Robot Learning. PMLR, 2023.

[6] Liang, Zhixuan, et al. "Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[7] Zhang, Tianjun, et al. "Efficient Planning in a Compact Latent Action Space." The Eleventh International Conference on Learning Representations

6. About additional related works (1/2)

Thanks to R-XZDy’s suggestion, we added in the following a discussion about suggested additional related work to highlight the novelty of our method. While it is accurate that previous research has explored discrete planning over learned structures, our work uniquely integrates these concepts within the goal-conditioned offline reinforcement learning (RL) framework—a context that, as rightly pointed out by the reviewer, has not been directly addressed in prior studies.

Specifically, although several cited works utilize VQ-VAEs to define landmarks or similar constructs, none of these approaches incorporate a time contrastive loss, the utility of which we demonstrate comprehensively in subsection 5.4 but also in appendix D.4. This contrastive loss plays a crucial role in enhancing the temporal coherence of the learned representations, thereby improving the effectiveness of our hierarchical planning mechanism.

Furthermore, our introduction of explicit trajectory stitching as a form of data augmentation is, to our knowledge, novel within the realm of offline GCRL. While alternative planning methods, such as shortest path algorithms over graphs [3], offer advantages in terms of speed and explainability, our sequence generator approach provides distinct benefits tailored to offline learning scenarios. Unlike shortest path methods, our approach prioritizes paths that are more aligned with the dataset, thereby enhancing success rates by discouraging routes where data is sparse through effective token prediction.

Below, we provide a detailed comparison paper by paper to elucidate the distinct contributions of our work:

1. Choreographer [1] (added to the related work)
   • Overview: Choreographer employs a VQ-VAE to discretize states into discrete ‘skills’. Policies are trained to reach states corresponding to their assigned skill-conditioning codes.
   • Comparison: While both Choreographer and our work utilize VQ-VAEs for state discretization, our approach introduces a time contrastive loss that enhances temporal coherence by encouraging temporally close states to share the same tokens. This results in more robust and meaningful landmark representations, which are crucial for effective hierarchical planning in offline settings. Additionally, Choreographer focuses primarily on skill discovery without integrating explicit trajectory stitching, which our method leverages to improve data augmentation and planning robustness.
2. Dr. Strategy [2] (added to the related work)
   • Overview: Dr. Strategy uses a VQ-VAE to define landmarks and conditions policies on these landmarks, primarily within an online RL framework.
   • Comparison: Our method extends the concept of landmark-based planning into the goal-conditioned offline RL setting, addressing challenges specific to offline data constraints. Unlike Dr. Strategy, which is tailored for online interactions, our approach incorporates explicit trajectory stitching and planning that aligns with data quantity, ensuring that the planned paths are supported by ample data, thereby enhancing reliability and performance in offline environments.
3. CQM [3] (Curriculum Reinforcement Learning with a Quantized World Model) (in the related work)
   • Overview: CQM employs a VQ-VAE to quantize the goal space, treats landmarks as vertices in a graph, evaluates edge costs using Q-values, and utilizes Djikstra’s algorithm for planning within an online RL framework.
   • Comparison: While CQM focuses on optimizing planning through Djikstra’s algorithm in an online context, our approach is specifically designed for offline RL. We prioritize planning paths that are well-represented within the dataset, leveraging our contrastive loss to ensure that our sequence generator predicts paths aligned with data availability. This strategy mitigates the risk of following suboptimal paths in low-data regions, a critical consideration in offline settings where data scarcity can significantly impact performance.
4. PTGM [4] (Pre-training Goal-Based Models for Sample-Efficient Reinforcement Learning)
   • Overview: PTGM utilizes simple k-means clustering for state discretization and applies this discretization after training a high-level policy in a semi-online manner.
   • Comparison: Unlike PTGM’s semi-online discretization, our method employs explicit trajectory stitching tailored for purely offline RL scenarios. This allows us to effectively augment the dataset by stitching trajectories, enhancing the robustness of our high-level planning without relying on online interactions. Furthermore, our use of contrastive loss provides additional robustness by ensuring temporal coherence in the discretized state space.

5. QPHIL performance with and without data augmentation

We thank the R-hhSM and R-mB11 for expressing their concerns. The proposed data augmentation serves to create longer trajectories than the initial ones by explicit trajectory stitching. However, those are not necessary in the antmaze-medium to antmaze-ultra datasets. Hence, data augmentation can create new unnecessary trajectories. The traditional D4RL antmaze environments used have very similar starting and goal positions for evaluation. Hence, because of the discretization, it is likely that our model converges to propose a good and unique trajectory which is sufficient to get very good performance. We created a variant of those environments called random-antmaze in section 5.5 where the starting state and the goal are selected randomly in the maze. In this setting, the difference between augmenting or not the data is no longer significant for the ultra mazes, while being essential in the extreme mazes. This suggests that the aforementioned performance disparity is due to the poor diversity of evaluation configuration of those traditional environments and that data augmentation is neutral for smaller-range settings and beneficial for longer-range settings.