Bridging the Gap Beteween SL and TD Learning via Q-conditioned maximization

1. The introduction lacks sufficient emphasis on motivation, such as the advantages and necessity of SL compared to TD under the goal-conditional setting. It would be better to discuss the importance of SL-based methods, like OCBC, in detail in the introduction.
2. Following Weakness 1, the experimental results also show a significant gap compared to TD-based algorithms (Table 1). It is still helpful to discuss this experiment phenomenon after Table 1.

The idea is not novel, basically a combination of tricks in exsiting literature. The expriments results can not support the title. See my questions for more details.

• Inconsistent and Incomplete Notations: The mathematical notations are poorly defined and inconsistent, for example, equations like Eq. (3), which omits necessary terms such as the expectation over the initial state distribution. These issues create significant barriers to understanding the proposed approach.

• Lack of Theoretical Rigor: Theorem 4.1 and its proof are presented in a sloppy and non-rigorous manner. Important terms are either undefined or unclear.

• Underwhelming Empirical Performance: The proposed method, GCReinSL, underperforms significantly compared to existing methods like IQL and CQL, particularly in the more challenging Antmaze datasets. The results fail to justify the claimed advantages of sequence modeling approaches over TD-based methods.

• I have spent a significant amount of time and effort thoroughly reviewing this paper, but the conceptual, theoretical, and empirical weaknesses, along with poor clarity, lead me to come to a conclusion that the paper is not ready for publication.

• Lack of Clarity: The paper is riddled with errors and unclear explanations, making it challenging to read (see below). The poor writing quality detracts from the overall presentation and makes it difficult to follow the core ideas.

Comments & questions

• Line 56-57: How come “the objective in goal-conditioned RL is equivalent to the Q-function”? How does a function “Q-function” is equivalent to an objective (either learning or optimization)? Perhaps, the authors intend to say “the objective function?” Objectives and objective functions are two different things.
• Line 67” What is “in-distribution Q-function?” Please properly define what you mean by “in-distribution” (perhaps with respect to offline data set?)
• Line 69: Is it “predicted Q-function” or “estimated Q-function”?
• Line 70: What is “the current maximum Q-function?” What do you mean by current? By “maximum Q-function” do you mean max of Q-functions or max_a Q(s,a)? The latter is NOT maximum Q-function… is maximum Q-value.
• In Eq.(3), expectation with respect to initial state distribution is NOT included in $J(\pi)$? Then, shouldn’t $J(\pi)$ be also function of s? Also, in Eq.(2), when you define conditional “discounted state occupancy distribution,” $\pi$ is used as a goal-conditioned policy $\pi(a, | s, g)$ is conditioned on a pair of state and goal. Now, in (3), the authors are using trajectory-wise policy in (3), particularly the conditional distribution within the expectation of (3), which is not consistent with the definition of (2) The authors should avoid any unnecessary overloading, and be clear about what $\pi$ they use. Perhaps, the authors need to appropriately define the relationship between $\pi(\tau | g)$ and $\pi(a, | s, g)$.
• In any real-world environment, how does an agent observe rewards as “the (exact) probability of reaching the goal at the next time stee” as defined in Eq.(4)? Do the trajectories defined in 171-172 contain these probability rewards as offline data?
• The conditional distribution “p^\pi_+(s_{t+} = g | s_0 = s,a)$ in Eq.(6) conditioned on state and action pair has been properly defined.
• Line 216-217, what is $\hat{Q}_t$? Is it $\hat{Q}_t = \hat{Q}(s_t, a_t)$? Please be precise.
• Line 216-219, the explanation is not clear. For example, the authors explain that OCBC methods will reach the state $g_1$ rather then $g$ since since Q-function is still zero. But, there are no explonations as to why… They say that “$\hat{Q}_t = 1$ is impossible to obtain given $\hat{Q}_0 = 0$. Please clearly explain why.
• Line 235: When introducing the “Latent Variable Model,” please define what $\psi$ is.
• What is $p(z|s)$ in Eq.(8)? The authors only mentioned $p(z|s,a) = \mathcal{N}(0,I)$ as a prior.
• Line 246-247: In “we can approximate the probability $p^{\pi(\cdot | \cdot ,g)}(s_{t+} = g | s, a)$ in Eq.(6) by $−\mathcal{L}_\text{ELBO}$”, how?
• What is $t$ in the expectation $E_t$ in Eq.(10)? There is NO mention of $t$ in the entire section of 4.3.
• In Theorem 4.1, after defining $**SG**= (s, g, a, Q)$, the authors write $Q(**SG**, a)$ for $Q_\text{max}$. So, $Q(**SG**, a) = Q(s, g, a, Q, a)$? At this point (along with the previous unclear presentation), the paper becomes very difficult to read…
• In Theorem 4.1, so $\mathbf{Q}^m$ is a policy?
• In Theorem 4.1, $\pi_{\theta}^* = \arg \min \mathcal{L}^m_Q$ what is the $\arg \min$ over? All of sudden, the authors introduce $\theta$ which I suppose is the parameter for the policy, then they need to define it. Also, if $\pi_{\theta}^*$ is function of $\theta$, but the loss $\mathcal{L}^m_Q$ does not depend on the parameter of a policy? The authors are very sloppy about the notations and presentations throughout the paper, yet even one of their main results (Theorem 4.1) fails to provide meaningful contributions particularly with its non-rigorous presentation.
• In the proof of Theorem 4.1 in Appendix A.2, aren’t $\mathbf{Q}^m$ a vector (or even matrix)? How do you define inequality between vectors, is it element-wise? Then, the authors should be explicit about that. What do you mean by “all Q-values from the offline dataset” in 917? All true Q-values? The proof of Theorem 4.1 makes difficult to validate its claim.
• After reading the proof of Theorem 4.2 in Appendix A.3 and also considering the statement itself, I do not think Theorem 4.2. is a proper mathematical theorem. “Remark” (or corollary at best) would be an adequate category.

Minor errors

Line 23: “by incorporating maximize” => “by incorporating maximizing?” Line 53 Acronym “DT” is used without properly defining what it is first. Line 54-55: The citation “Zhuang et al. (2024)” should be used with \citep Line 75: Acronym “RvS is used without properly defining what it is first. Line 145: “maximise” or “maximize” like the other expressions in the text? It would be good to maintain the style consistency. Line 162: “Q-function are” => “Q-functions are” Line 173: Perhaps, “In each $\tau_i$ for $i in 1, …, N$“ would be more clear. Line 177: “provide present” => “present” Line 220: “OOD” => “out-of-distribution (OOD)” please define acronyms when first introduced. Line 262: “After estimate the Q-function” => “After estimating the Q-function” Line 274-275: “more weights to the $Q$ larger than” is missing $\hat{Q}$.

• The method needs to learn a conditional variational autoencoder which could introduce additional computational overhead and complexity.
• The proposed method is constrained by the goal-conditioned formulation which may limit its application.