Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning

We sincerely thank Reviewer qrJ2 for the positive comments on the motivation of the proposed method and “order consistency” evaluation metric. We also appreciate your constructive feedback regarding theoretical justifications, terminology, reward assumptions, etc. Below, we address your concerns in detail.

Weakness 1: Theoretical justification for the OTA value

We thank the reviewer for these important questions. (1) The value function in our formulation can be interpreted as the expected sum of discounted rewards—i.e., the return—which aligns with the standard definition in reinforcement learning. (2) Our approach builds on the options framework, where it has been thoroughly studied that the value function converges to the optimal value under Bellman recursion. For a more detailed theoretical proof, we kindly refer the reviewer to foundational work [1].

Question 1: Clarification of “reward-free dataset”

We agree that the term "reward-free" may be misleading without clarification. In our setting, the offline dataset used for GCRL training consists of trajectories in the form of $(s_0, a_0, s_1, a_1, \dots)$, and does not include explicit reward signals. However, during training, we apply a sparse binary reward function using hindsight experience replay (HER) [2]. Thus, while the original dataset itself is unlabeled in terms of rewards, rewards are introduced during training via relabeling. To avoid confusion, we will replace the term "reward-free dataset" with "reward-unlabeled dataset”.

Question 2: Clarification of “policy extraction”

We use the term “policy extraction” to refer to the process of deriving a policy from a learned value function, a usage consistent with prior works such as IQL [3], OGBench [4], and JaxGCRL [5]. This term emphasizes the decoupling between value learning and policy learning, particularly in algorithms like AWR, which IQL builds upon. Specifically, we will make the following revisions:

• Line 29 will be removed for clarity and conciseness.
• In earlier sections (e.g., lines 49, 52, 60, ...), where we refer to general policy learning, we will consistently use policy learning instead of policy extraction.
• Beginning with the related work section (e.g., line 123), where AWR is discussed explicitly, we will adopt the term policy extraction and provide a brief definition for clarity.

Question 3: Reward function formulation

We agree that in continuous state spaces, a more appropriate reward formulation would be $r(s,g)=-\mathbf{1}(|s-g|^2>\epsilon)$. However, in practice, this formulation introduces certain challenges: (1) it introduces an additional hyperparameter $\epsilon$, (2) it increases the computational overhead in HER-style goal relabeling, especially for high-dimensional states. For these reasons, we chose to use a sparse binary reward of $-\mathbf{1}(s\neq g)$ in our experiments following other GCRL benchmarks [4,5]. We will clarify this design choice in the experimental section of the revised manuscript.

Question 4: Over-optimistic value estimates in Figure 2

Thank you for your insightful observation regarding over-optimistic estimates of the high-level value function when $s$ is far from the goal $g$. We hypothesize that this behavior arises from the value learning mechanism in IQL, which employs expectile regression to estimate the upper expectile of TD targets. As the distance between $s$ and $g$ increases, the variance (noise) of the TD targets also tends to increase (Figure 8 in HIQL [6]). Consequently, the upper expectile can become significantly biased, leading to overestimated values for distant goals. That is, the estimation error tends to accumulate over longer horizons in IQL, resulting in increasingly over-optimistic value estimates. In contrast, our OTA value learning approach effectively mitigates this issue, even as the horizon increases. We will incorporate this explanation in the revised manuscript and supplement it with additional visualizations comparing OTA with the ground-truth value function to support the claim.

Reference

[1] Sutton et al., "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning." Artificial intelligence 1999

[2] Andrychowicz et al., “Hindsight Experience Replay”, NeurIPS 2017

[3] Kostrikov et al., “Offline Reinforcement Learning with Implicit Q-Learning”, ICLR 2022

[4] Park et al., “Ogbench: Benchmarking Offline Goal-Conditioned RL”, ICLR 2025

[5] Bortkiewicz et al., “Accelerating Goal-Conditioned RL Algorithms and Research”, ICLR 2025

[6] Park et al., “HIQL: Offline Goal-Conditioned RL with Latent States as Actions”, NeurIPS 2023

Thanks for your replies. I have the following comments:

Weakness. I would suggest that the authors clarify this point and cite the relevant work in the section.

Q1. It would be better to remove reward-free or reward-unlabelled. Since the reward function is provided in offline GCRL, it is misleading to describe it as reward-free or reward-unlabelled.

Q3. I don't think the two challenges are valid. (1) It does not introduce an additional hyperparameter — even if you don’t use my suggested definition, the threshold $\epsilon$ still appears in your code. (2) For the same reason — even without using my definition, you still need to compute the norm to obtain the reward after goal relabelling.

Although these are minor issues, the authors should address them seriously.

Thanks for your reply. I'm happy to discuss with you to clarify this issue. I'll update my score accordingly.

Additionally, you don't need to add the experiments to compare the norm and the exact-match. It makes sense to use exact-match during goal-relabelling.

We sincerely thank Reviewer qyqg for the positive comments on the motivation of our paper and the analysis experiments. We also appreciate your constructive feedback regarding the empirical evaluation, including hyperparameters, manipulation tasks, etc. Below, we address your concerns in detail.

Weaknesses 1: Hyperparameter tuning

Thank you for the comments on tuning the hyperparameters of OTA. We agree that we searched the hyperparameters extensively to report our results. To show that OTA also outperforms HIQL with the same hyperparameters (e.g., $k, \beta^h, \beta^{\ell}$) as HIQL, we also carry out additional experiments in complex maze tasks (i.e., large and giant mazes except for explore dataset). The table below shows the results with the same hyperparameters to HIQL except for $n$. In this table, the success rate of OTA is still much higher than HIQL, meaning that simply applying the temporal abstraction can improve the performance.

Weaknesses 2: Performance on play datasets

To demonstrate the effectiveness of temporal abstraction between action sequences in the presence of noisy data, we conducted experiments using a noisy dataset. Additionally, we evaluated the performance of OTA on play datasets. The results are summarized in the table below. For a fair comparison, we fixed the hyperparameters $(k, \beta^h, \beta^{\ell})$ for both OTA and HIQL and varied only $n=2$.

Weaknesses 3: Assumptions on the data optimality

First, we want to clarify that the update formula in Equation (4) does not assume data optimality. As in the definition of the option we used, we assume that the underlying policy is the behavior policy, which is used to collect the offline data, and the behavior policy is extremely suboptimal. Therefore, the option-aware value update in Equation (4) can be performed regardless of the data characteristics. In fact, we observed significant performance improvements on the extremely suboptimal datasets such as antmaze-medium-explore and antmaze-large-explore.

Minor issue: Option description

Thank you for your comments on the notion of options we used. As the reviewer pointed out, we do not use action chunking, but the above explanation in the manuscript may induce misunderstanding. We will revise the explanation in the camera-ready version.

We sincerely thank Reviewer x6Va for the positive comments on our analysis of the failure cases of HIQL and the strength of our experimental results. We also appreciate your constructive feedback regarding the limitations in stochastic dynamics and the relationship to n-step TD learning, etc. Below, we address your concerns in detail.

Weakness 1: Limitations inherited from IQL

As the reviewer has mentioned, our method and analyses were developed under the assumption of a deterministic environment. We should have stated this point more clearly, especially when introducing concepts such as "order inconsistency" and the notion of an "optimal trajectory”, which only make sense in deterministic environments. We will revise the manuscript to clarify this assumption. While our method builds on IQL due to its simplicity and strong empirical performance, we do acknowledge that IQL has known limitations in stochastic environments.

However, we note that this assumption is not unique to our work; several baseline methods [1,2,3] used in the offline GCRL benchmark also focus on deterministic environments, primarily due to the challenging nature of offline GCRL problems in more general stochastic environments. To that end, while our framework inherits certain limitations from IQL, we believe it still makes a significant contribution in deterministic environments by enabling objective comparisons with prior work under a consistent evaluation setting. Furthermore, we believe the core ideas of our approach (i.e., the option-aware target) could be extended to other offline RL algorithms that are more robust to stochastic environments [4,5], which we consider a promising direction for future work.

As the reviewer pointed out, expectile loss may lead to over-optimistic value estimates in stochastic dynamics, using the $n$-step forward states for updating the value may exacerbate the bias. However, in deterministic dynamics, our experiments show that multi-step returns reduce bias and improve performance, balancing the bias-variance trade-off.

Weakness 2: Link to the original options

• Non-Markov termination condition

Thank you for pointing out that our use of a fixed n-step timeout is a non-Markovian termination condition. We would like to clarify that the original options framework by Sutton et al. [6] is general enough to accommodate such cases through the use of semi-Markov options.

• Options as high-level choices

While options can represent high-level choices when learned explicitly, our setting differs: we leverage them implicitly from the dataset. The behavior policy $\mu$ used during data collection consists of various high-level policies (e.g., grasping objects, moving items). Therefore, depending on which trajectories are sampled from the dataset, appropriate high-level choices are utilized, enabling temporal abstraction without explicit option learning.

• Aggregate discount $\gamma^n$

Several works, including the original options paper [6], apply a discount factor of $\gamma^n$ to rewards after $n$ steps. However, some studies [7,8,9] use option-level discounting when learning the value function. In our case, we adopt the option-level discounting scheme to enable temporal abstraction through an $n$-step target.

Weakness 3: Connection with $n$-step TD learning

The exact formula of TD target for $n$-step and OTA is as follows:

• $n$-step target: $\sum_{i=0}^{n-1} - \gamma^i \cdot 1(s_i \neq g) + \gamma^n V_\theta (s_{t+n}, g)$
• OTA target: $-1(s^{\Omega}\neq g) + \gamma V_\theta (s^{\Omega}, g)$

In the above formula, since both targets mainly utilize n-step forward value, one may think both formulas are quite similar. However, we want to clarify that extending the n-step target to the OTA target is not as straightforward. Rather, temporal abstraction through the option framework provides a natural explanation for the insights gained in our motivation experiments.

Though the reward signal has a minor difference (i.e., the scale of rewards), the critical difference between $n$-step TD and OTA comes from the choice of $\gamma$, which controls the information decay ratio as we perform the TD update. Since the standard $n$-step target typically uses the same $\gamma$ as in 1-step target, the discount factor applied to the value function across the trajectory remains the same as in the 1-step TD. Or, equivalently, the discount factor applied to the value function in the n-step target decays exponentially with $n$ (e.g., $\gamma^n \approx 0.95$ for $(n,\gamma)=(5,0.99)$ or $(n,\gamma)=(10,0.995)$) while that in the OTA target is independent of $n$.

To demonstrate that the excessive decay of the discount factor in the standard $n$-step target hinders learning of the order-consistent value, we carried out additional experiments comparing the order consistency ratio ($r^c$) of value functions learned by 1-step TD, $n$-step TD, and OTA, all using the same discount factor $\gamma$. We also used the same $n$ for $n$-step TD and OTA. In the table below, in contrast to OTA, we clearly observe that the $r^c$ for $n$-step TD has no improvement over 1-step TD since using the $n$-step TD target may still suffer from the estimation error of the value function due to the same level of discount factor.

The above results suggest that one may try to improve n-step TD by controlling $\gamma$ appropriately for each $n$ so that the discount factor can be adjusted. However, it is clear that such an approach would introduce additional complexity in hyperparameter selection. In contrast, our OTA fixes $\gamma$ regardless of $n$, which makes the scheme much simpler.

In the revised manuscript, we will clarify the connection between OTA and $n$-step TD learning. In particular, we will address the clarity feedback by summarizing some of the detailed explanations in the Introduction and adding a discussion of the similarity to n-step TD in Section 5.

Weakness 4: Small performance improvement in manipulation tasks

We analyze the order consistency ratio ($r^c$) of $V^h$ in HIQL on cube-single-play and cube-double-play, both achieving a high ratio of 0.98. The reason for achieving high $r^c$ in HIQL is the horizon of both tasks is relatively small (less than 500) compared to complex maze tasks. We, therefore, conclude that in the case of short-horizon manipulation tasks, the core problem lies in the policy extraction scheme for high-level policy (i.e., AWR for $\pi^h$), not in the value estimation. However, we believe that for more complex and long-horizon robot manipulation tasks in which 1-step TD cannot learn order-consistent value, OTA can improve the performance of HIQL.

Minor comment: Clarification of "noisy" and "macro-action"

Regarding line 186, we agree that the term "noisy" can be ambiguous. Here, it refers to high estimation error between predicted and true values. As described in HIQL (see Figure 8 in HIQL), the variance of the value function increases with longer horizons, destabilizing value learning and resulting in larger value errors, as illustrated in our Figure 2. We will clarify this explanation in the related work section and revise the wording.

For line 194, we appreciate the distinction you pointed out between options and macro-actions. Since we define options along with the termination conditions from the offline dataset, they do not necessarily have fixed lengths. Therefore, the term "options" is more appropriate in our setting, and we will update the manuscript accordingly.

Question 1: Importance of sign matching

For high-level policy learning, the advantage is computed by $V^{h}(s_{t+k}, g) - V^{h}(s_t,g)$. With large $k$ values (e.g., 25 in antmaze, 100 in humanoidmaze), this results in large-magnitude advantages. If the sign of the advantage is incorrect, the magnitude of regression weights (which is the exponentiated advantage) is drastically increased or decreased, causing unintended improper subgoal regression for high-level policy. For example, an advantage range of $[-1, 1]$ with $\beta^h=3$, the regression weights vary from $e^{-3}\approx 0.05$ to $e^{3}\approx 20$, which is significantly far from 'just negative' or 'just positive'. In our experiment, the magnitude of the advantage often exceeds 10, and we also use $\beta^h>1$, making the impact of sign mismatch even more critical.

While small-magnitude advantages may be less affected by sign errors, a low advantage magnitude implies that the objective becomes similar to behavior cloning, which does not give any useful information on learning proper subgoals. For this reason, we first scale $\beta^h$ to ensure sufficient advantage magnitude, and then apply AWR.

Reference

[1] Ghosh et al., “Learning to reach goals via iterated supervised learning”, ICLR 2021

[2] Wang et al., “Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning”, ICML 2023

[3] Park et al., “HIQL: Offline Goal-Conditioned RL with Latent States as Actions”, NeurIPS 2023

[4] Villaflor et al., “Addressing optimism bias in sequence modeling for reinforcement learning”, ICML 2022

[5] Liu et al., “A Tractable Inference Perspective of Offline RL”, NeurIPS 2024

[6] Sutton et al., "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.", Artificial intelligence 1999

[7] Klissarov et al., "Flexible Option Learning", NeurIPS 2021

[8] Nachum et al., "Data-Efficient Hierarchical Reinforcement Learning", NeurIPS 2018

[9] Chai et al., "MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions", ICLR 2025

We sincerely thank Reviewer 1VvB for the positive comments on our analysis of the bottleneck in hierarchical policies and the simplicity of our method. We also appreciate your constructive feedback regarding the additional hyperparameter $n$ and the guarantee of complete orderliness, etc. Below, we address your concerns in detail.

Weakness 1: Additional hyperparameter $n$

We agree that introducing another hyperparameter $n$ may increase the complexity of parameter tuning. However, we would like to clarify that the search complexity of our method is not prohibitively large. Instead of performing an extensive hyperparameter sweep from scratch, you can follow the guidelines in Question 3 and attain strong performance within a much narrower search space. We hope this addresses your concern.

Weakness 2: A limitation in guaranteeing complete order consistency

We agree that our method does not fully guarantee perfect value ordering. Nevertheless, we would like to emphasize that our simple approach significantly reduces value‑ordering errors and, in turn, achieves much higher performance compared to the baselines.

Question 1: Impact of step $n$ on short-horizon value estimation

The objective of the high-level policy in long-horizon planning is to identify reachable subgoals that the low-level policy can achieve within a short-horizon. Accordingly, the high-level value function is only used for long-horizon planning. In contrast, training the low-level policy requires accurate value estimation within short horizons.

That is, short-horizon value estimation is utilized solely for training the low-level policy through the low-level value function, whereas the temporal abstraction step $n$ affects only the high-level value function. Therefore, adjusting the temporal abstraction step $n$ turns out not to have a direct impact on the value estimation of short-horizon states.

Question 2: Separate value functions with different step sizes for each policy

We apologize for the lack of clarity in the manuscript and appreciate the reviewer’s attention to this point.

Indeed, we agree with the reviewer and already follow this design in our approach. Specifically, unlike HIQL which trains a single value function $V$ for both $V^h$ and $V^\ell$, we train separate value functions for the high-level and low-level policies, each with a different step size -- i.e., the low-level value function, $V^\ell$, is trained using a standard 1-step target, while the high-level value function, $V^h$, uses an $n$-step option-aware target. We will further clarify this point in the final version of the paper.

Question 3: Automatically selecting optimal $n$ and $k$

While devising more automated strategies for selecting hyperparameters could be further explored, we have found that narrowing the search to a small, dataset‑ or environment‑specific range of plausible values works well in practice. For example, our OTA can perform effectively on short-horizon tasks such as AntMaze (≈ 1000 execution steps) with $(n,k) = (5,25)$, whereas it performs well on long-horizon tasks like HumanoidMaze (≈ 4000 execution steps) with $(n,k) = (20,100)$. This rough linear scaling trend of $(n,k)$ with respect to the execution steps implies that having prior knowledge about the specific execution steps of the dataset/environment can quickly narrow down the range of the relevant hyperparameters so that full exhaustive searches can be avoided.

Thank you for the response. I will keep my score