Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction

General Response: We sincerely thank the reviewer for taking the time to provide detailed feedback and for raising these concerns. We sincerely apologize for any misunderstanding caused by our writing, and we greatly appreciate the opportunity to clarify the differences between the two papers and address the reviewer’s comments in detail.

Comment 1: After reading both, I believe that the papers are so similar that they should not be two separate submissions. For example, the challenges they aim to solve in hierarchical RL are identical.

Response to Comment 1: We acknowledge that both DIPPER and HPO papers share a similar motivation, as both aim to address challenges in hierarchical reinforcement learning (HRL) using ideas from preference optimization. However, we would like to highlight the key differences between the two. While both tackle non-stationarity and infeasible subgoal generation in HRL, the solution approach, derivations, and practical implementations are distinct. Below, we provide a detailed breakdown to address these concerns further.

Comment 2: The resulting "novel" solution for the procedure proposed is in Eq 13 in HPO paper and Eq 15 in DIPPER other, and are exactly identical.

Response to Comment 2: We understand the concern about the similarity in the mathematical structure of the two equations. However, the underlying derivation, assumptions, and implementation of these equations are different. Below, we provide a side-by-side comparison of the two equations and highlight the key differences:

Equation (15) from the DIPPER is given by:

$

\mathcal{L}^d = - \mathbb{E}_{(\tau^1, \tau^2) \sim \mathcal{D}} \bigg[& \log \sigma \bigg( \sum_{t=0}^{T-1} \big(\alpha \log \pi_U \big( g^1_t \mid s^1_t\big) - \alpha \log \pi_U \big( g^2_t \mid s^2_t\big) + \lambda \underbrace{{(V_{L}^{k}(s^1_t, g^1_t) - V_{L}^k(s^2_t, g^2_t) )}}_{A:=} \big) \bigg) \bigg]. \tag{15}

$

Equation (13) from the HPO paper is given by:

$

\mathcal{L}(\pi^H_{\star}, \mathcal{D}) = - \mathbb{E}_{(\tau^1, \tau^2, y) \sim \mathcal{D}} \bigg[& \log \sigma \bigg( \sum_{t=0}^{T-1} \big(\beta \log \pi^H_{\star} \big( g^1_t \mid s^1_t, g^{\star} \big) - \beta \log \pi^H_{\star} \big( g^2_t \mid s^2_t, g^{\star} \big) + \lambda \underbrace{(V_{\pi_L}(s^1_t, g^1_t) -V_{\pi_L}(s^2_t, g^2_t))}_{B:=} \big) \bigg) \bigg]. \tag{13}

$

Key Differences between Eq. (15) and Eq. (13): To highlight the difference between Eq. (13) and Eq. (15), let us consider the terms A in Eq. (15) and B in Eq. (13). We remark that the value function used in A is $V_{L}^k$ is a k step approximation of the optimal value function (which is derived using huristics in DIPPER without rigorous mathematical justifications, but works in practice, and also requires higher value of $k$). This leads to a double-loop algorithm proposed in DIPPER. One loop is to obtain the value of $V_{L}^k$ (k iterations), and then another loop is to update the policy after calculating the gradient of Eq. (15).

In contrast, let us consider the term B in Eq. (13) from HPO; we note that it just has the value function evaluations $V_{\pi_L}$ without any additional inner loop to calculate the optimal value function, which is required in Eq. (15). Therefore, the algorithm proposed in HPO is a single loop algorithm to solve the challenges of HRL.

We concede that both approaches are trying to solve the challenges posed by HRL, but the solution approaches are different, which leads to different algorithms (two loop algorithm to solve Eq. (15) and only single loop algorithm to solve Eq. (13)). We agree with the reviewer that the contributions can be incremental, but we humbly request the reviewer not to raise an ethics flag because that was never the intentions and we extremely apologies if our writing has lead to the wrong impression.

Additional Fundamental Difference between DIPPER and HPO.

(i) DIPPER requires access to a reference policy $\pi_{ref}$ as mentioned in the objective in [Eq. (5), DIPPER]. Also the derivation and the loss function derived for DIPPER in Eq. (15) holds only for the specific design choice of the reference policy defined in [Eq. (9), DIPPER], which depends upon the optimal value function, which later requires a k-step approximation of the optimal value function.

(ii) On the other hand, the algorithmic development in HPO is independent of any reference policy and does not require any such assumptions. The derivations in HPO are motivated by the developments in this paper (https://arxiv.org/pdf/2404.12358).

Comparing EQ 15 and Eq 13, difference comes in $k$ step Value function

This $k$ step value function is, for all practical purposes, a design choice. Ironically, Algorithm 1 of this paper, HPO, line 8, literally uses $V_\pi^k$ which is only in the DIPPER paper; this signifies essentially a direct copy-paste of the latex algorithm block from DIPPER to HPO's algorithm. I'm not convinced that $k>1$ is even relevant as the DIPPER paper does not list $k$'s value in the hyperparameter list.

Needing $\pi_{\text{ref}}$

This reference policy is absorbed into the objective, and again, results in very little difference between the two algorithms.

Thus I am unconvinced and will be keeping my ethics flag. If the ethics review sees nothing wrong, I will be reviewing both papers as standalone contributions. But, at least to me, submitting two nearly identical papers with the same claimed contributions and nearly identical algorithms/objectives that have nearly identical experimental results seems to be ethically flawed.

I would suggest combining into one paper and resubmitting next time.