Conservative Offline Goal-Conditioned Implicit V-Learning

Thank you for your thoughtful review and valuable suggestions. We have carefully addressed each of your concerns in the responses below.

R1: Methods and Evaluation Criteria

We have extended our evaluation to manipulation environments (see Table 1 in the linked PDF for details). Detailed analysis is provided in our response R1 to Reviewer 96R5.

R2: Supplementary Material

We would like to polish the README file to ensure our algorithm can be easily reproduced.

R3: Weaknesses

1. We have conducted additional experiments comparing CGCIVL’s performance with and without the conservative regularization term (i.e., $\eta\neq 0$ and $\eta=0$ respectively). As shown in Figure 1 (see the linked PDF for details), removing this component leads to a significant performance drop, confirming its critical role. Furthermore, we observe that the performance is robust within a suitable range of the regularization coefficient, but both excessively small and large values degrade the results.
2. See our response in R1.
3. In our paper, all algorithms were evaluated under the same number of training steps to ensure a fair comparison. Different from results in OGbench where baselines were trained with 1M steps, we trained all algorithms for longer steps in complicated environments (See Appendix C.3 for details). Figure 3 (see the linked PDF for details) shows training curves for all algorithms in these complex environments.
4. We would like to clarify that CQL and quasimetric are the key techniques of our algorithm, addressing value overestimation on unconnected state-goal pairs. IQL is a necessary policy improvement technique, which could be replaced with other methods. The hierarchical structure is a common approach to handle long-horizon tasks and may be omitted in non-long-horizon environments.

R4: Comments

1. $\mu(g|s)$ denotes an arbitrary distribution which satisfies $\operatorname{supp} \mu \subset \operatorname{supp} p_{m}^{\alpha}$. Trajectory truncation only alters the goal associated with states in different segments, without directly changing the distribution of states in the dataset. Therefore, we can approximate sampling states from $\mathcal{D}$ as sampling from $d^{\pi_{\beta}}$.
2. We'll mention $V_{\theta_d}\in\mathcal{Q}^-(S)$ near Eq. (13) for clarification in the revised manuscript.
3. The parameter $\beta$ in Equations (14)-(15) serves as the temperature coefficient for both the high-level and low-level policy extraction. We empirically set $\beta=3.0$ across all experiments.

We sincerely appreciate your time and effort in reviewing our work. We have addressed each concern you raised below.

R1: Claims and Evidence

Thank you for the reviewer’s comment. The theoretical guarantees mentioned in our paper indeed refer to the algorithm prototype based on Eq. (8), rather than the practical algorithm. However, in Lemma 1 (Appendix A), we prove that the expectile regression in the practical algorithm is equivalent to the Bellman operator in Eq. (8). Additionally, compared to Eq. (8), the practical algorithm incorporates two key techniques: 1) hierarchical learning to address long-horizon tasks, and 2) quasimetric distillation to improve the efficiency of value learning. These techniques do not fundamentally alter the core components of the CQL-inspired penalty term and quasimetric, which form the foundation of the theoretical analysis. Therefore, while Eq. (8) does not exactly match the practical algorithm, we believe the practical algorithm still benefits from the theoretical guarantees established by the analysis.

R2: Theoretical Claims

1. We clarify notations you mentioned as follows:
   • $\mu(g|s)$ denotes an arbitrary distribution which satisfies $\operatorname{supp} \mu \subset \operatorname{supp} p_{m}^{\alpha}$.
   • $\hat{V}$ represents an empirical estimate of the true value function $V$ during iteration.
   • $\hat{B}$ denotes the empirical Bellman operator, which is the sample-based counterpart of the theoretical Bellman operator.
2. Yes. The correct statement of Proposition 4.5 should be: For any $\epsilon>0$ and $\eta>0$, there exists a hyperparameter $\alpha$ such that the inequality holds.

R3: Relation to Broader Scientific Literature

We would like to clarify that our work extends beyond a mere combination of existing techniques. The key contributions are:

• Problem identification:

  To the best of our knowledge, we are the first to formalize the critical issue of value overestimation for unconnected state-goal pairs in offline GCRL.

• Feasible solutions:

  Our solution penalizes the values of all cross-trajectory state-goal pairs while ensuring that values on connected pairs are not excessively under-estimated. We introduce a CQL-inspired regularization term for the first goal and use a quasimetric model for accurate value estimation of connected pairs to achieve the second. Both methods are supported by theoretical guarantees.

• Difference with original methods

  Unlike CQL, which penalizes OOD actions, we introduce a penalty term tailored for state-goal pairs. Unlike QRL, which trains value functions without value iteration, our approach incorporates quasimetric properties into the value iteration process to ensure accurate value estimation for connected state-goal pairs.

The novelty of our method lies in re-engineering these components to address a new problem in offline GCRL, rather than merely combining them.

R4: Weaknesses

1. The originality of our work is discussed in R3.
2. In R2.1, we provide explanations for any undefined notations and will include these details in the revised version of the paper. As defined in Sec 2, in Algorithm 1, states are randomly sampled, and goals are sampled in two ways: 1) $p_{rand}^{\mathcal{D}}(g)$ samples uniformly from all states in $\mathcal{D}$, and 2) $p_m^{\alpha}(g|s)$ samples from the same trajectory as state $s$ with probability $\alpha$, otherwise using $p_{rand}^{\mathcal{D}}(g)$.

R5: Other Comments or Suggestions

1. See R2.1.
2. See R4.2.
3. Future revisions will implement distinct notation per proposition for clarity.

R6: Questions

1. Although the proof in the original paper is based on discrete state and action spaces, its key components can also be extended to continuous settings. Non-negative penalty terms for underestimation can be generalized to density-based terms, and concentration bounds for the empirical Bellman operator $\hat{B}^{\pi}$ do not require discretization. The Neumann series ensures that $(I - \gamma P^{\pi})^{-1}$ remains well-defined in continuous spaces when $\gamma < 1$. Therefore, while Theorem 4.1 has not been strictly proven for continuous settings, our algorithm still benefits from the theoretical analysis in continuous environments, as further supported by experimental results.
2. As described in Sec 2 of our paper, the distance between state and goal should satisfies properties of quasimetric (Eq. (6)). However, the value function should exhibit an inverse relationship with the distance away from the goal (Eq. (7)). Thus we have $V(A;B)\geq V(A;C)+V(C;B)$, which is hold when $V(A;B)=0, V(A,C)<0,V(C,B)<0$.
3. Please refer to R1 where we discuss the differences between the practical algorithm and Eq. (8).
4. Proposition 4.5 is based on Theorem 4.1 by incorporating the quasimetric and replacing $\mu$ with the uniform distribution $p_{rand}^{\mathcal{D}}$. Consequently, Proposition 4.5 cannot hold if the conservatism is not included.

We sincerely appreciate your insightful feedback on our work. Please refer to detailed responses below to each of the raised concerns.

R1: Weaknesses

In order to provide more comprehensive evaluation of the performance, we have conducted additional experiments in three manipulation environments (Cube, Scene, Puzzle) - comprising a total of 8 manipulation tasks of varying complexity. The results, presented in Table 1 (see the linked PDF for details), demonstrate that CGCIVL achieves superior performance in all manipulation tasks, particularly in scene and elementary cube tasks, consistent with results in maze environments. We will include these additional experimental results in the revised version of the paper.

R2: Questions

1. In the original paper, we aimed to validate the algorithm in solving the goal stitching task. However, OGbench currently only provides the stitch dataset in maze environments. Nevertheless, we have supplemented our evaluation with additional experiments in manipulation environments to further verify the algorithm’s performance (See our response in R1). We will expand the experimental scope and incorporate these additional results into the final version of the paper.
2. Besides the analysis of $\alpha$, we have also performed sensitivity studies on both the penalty coefficient $\eta$ and the subgoal interval $k$. The analysis of $\eta$ and related experiments can be found in our response R2 to Reviewer Q24N. Figure 2 (see the linked PDF for details) shows the the performance of CGCIVL across different subgoal interval sizes. Results indicate that CGCIVL achieves the optimal performance with $k$ between $25$ and $50$. Overly small values of $k$ lead to the "signal-to-noise" issue in the value functions, as identified in the HIQL paper[1], while excessively large values of $k$ makes subgoals difficult to be achieved.

R3: Other Comments or Suggestions

We will include the mean score for each environment in the revised version of our paper to provide more comprehensive results.

[1] Park, S. , Ghosh, D. , Eysenbach, B. , and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems,36, 2024b.

We sincerely appreciate the reviewer’s constructive feedback. Below, we respond to each concern point-by-point.

R1: Methods and Evaluation Criteria

The choice between using $V_{\theta_v}$ or the distilled $V_{\theta_d}$ for advantage estimation ($\tilde{A}^h$ and $\tilde{A}^h$) is flexible, as both approaches are empirically valid, surpassing all baselines. Our experiments confirm that $V_{\theta_d}$ achieves comparable performance when used for policy extraction, suggesting either value function can be adopted without compromising results. We will clarify this point in the final paper.

R2: Theoretical Claims

Thank you for pointing out this question. Here we briefly explain why there exists an $\alpha$ satisfies condition for all state-goal pairs. Proof of Proposition 4.3 demonstrates that for any fixed $\epsilon > 0$ and arbitrary tuple $x=(s^+,s^-,g)$ sampled from dataset, where $(s^+,g)$ is in-trajectory and $(s^-,g)$ is cross-trajectory, there exists an $\alpha_x\in(0,1)$, such that inequality holds when $1>\alpha>\alpha_x$. Consequently, let $\tilde{\alpha}=\sup_x{\alpha_x}$, then we could find a static $\alpha\in (\tilde{\alpha},1)$ , which satisfies the condition for all state-goal pairs. We will provide further clarification in the revised version of the paper.

R3: Experimental Designs or Analyses

As suggested, we have conducted additional ablation studies to analyze the sensitivity of the penalty coefficient $\eta$, and results are presented in Figure 1 (see the linked PDF for details). The results indicate that both excessively small $\eta$ (causing insufficient regularization) and excessively large $\eta$ (over-constraining the optimization) degrade the performance. In practice, we determine the optimal $\eta$ through empirical validation across multiple candidate values.

R4: Other Comments or Suggestions

Maximizing values for connected state-goal pairs is indeed an interesting direction. However, we might need to carefully address several practical considerations: 1) directly sampling from the distribution of connected state-goal pairs in the dataset is difficult, and 2) further analysis is required to establish appropriate theoretical bounds, similar to those in CQL, when incorporating this approach. We plan to thoroughly explore these open issues in future work.