We thank the reviewer for the responses and suggestions for improving the paper. The reviewer brought up two main questions about hyperparameter selection and comparisons to Monte Carlo InfoNCE loss (contrastive RL (CPC)) in the experiments. We have attempted to address these questions by ablating different hyperparameters (new Fig. 10) and adding comparisons to contrastive RL (CPC) in online experiments (Fig. 2 and Appendix Fig. 6). Together with the discussion below, does this fully address the reviewer’s concerns?

How is the proposed method sensitive to hyper-parameters? Do we need careful hyper-parameter tuning to make it work? Is there any intuitive guidance about how to adjust the hyper-parameters?

We ran additional ablation experiments to study the effect of different hyperparameters for TD InfoNCE listed in table 2 and the discount factor $\gamma$. For each hyperparameter, we selected a set of different values and conducted experiments on $`push (state)`$ and $`slide (state)`$, one easy task and one challenging task, for five random seeds. We report mean and standard deviation of success rates in Appendix E.1. These results suggest that while some values of the hyperparameter have similar effects, e.g. actor learning rate = $5 \times 10^{-5}$ vs $1 \times 10^{-4}$, our choice of combination is optimal for TD InfoNCE. For fair comparison, we fixed the batch size of TD InfoNCE to 256, the same number as other baselines (Appendix E of [1] and Appendix C.3 of [2]).

Could you please also compare with the algorithm using Monte Carlo estimator of InfoNCE loss

As suggested by the reviewer, we ran new experiments comparing TD InfoNCE to contrastive RL (CPC) (MC InfoNCE) using the official contrastive RL repository [1]. We report results on the Fetch robotics benchmark in Fig. 2 and Appendix Fig. 6, showing mean and standard deviations over five random seeds. Note that we use contrastive RL to denote MC InfoNCE (contrastive RL (CPC)) and contrastive RL (NCE) to denote the original contrastive RL algorithm (See [1] for details). These results suggest that using InfoNCE loss for contrastive RL achieves a slightly better performance than using NCE loss for it on 4 / 8 tasks. Nonetheless, TD InfoNCE still outperforms this InfoNCE variant of contrastive RL (or Monte Carlo InfoNCE) and our conclusions remain the same. We have revised Sec. 4.1 to add this baseline.

For example, it is claimed that the proposed method "enables us to perform counterfactual reasoning". However, this point is not clear in the following section. Could you please explain it in detail?

We have revised this sentence to clarify what we mean by counterfactual reasoning (off-policy reasoning): (a) successfully stitching together pieces of different trajectories in an offline dataset to find a path between unseen state and goal pairs; (b) Finding a path that is shorter than the average paths demonstrated in the dataset. These two capabilities are demonstrated by our experiments in Sec. 4.4 and Appendix D.4.

In Algorithm 1, many notations are introduced for the first time without any definition. Could you please clarify them?

We have added notations in Algorithm 1 and defer details of the full goal-conditioned TD InfoNCE algorithm to Appendix D.1 due to limited pages.

Thank authors for the detailed response. I especially appreciate the additional experiment results, which help mitigate my concerns. I'd keep leaning toward acceptance.

We thank the reviewer for the detailed response and helpful suggestions for improving the paper. It seems that the reviewer’s main question is about the novelty of the paper. Our work does build upon prior work of contrastive RL and C-Learning. Contrastive RL is an on-policy Monte Carlo method, while our proposed method is a off-policy temporal difference version of it, achieving higher sample efficiency. C-learning can be interpreted as a TD version of a NCE (2-class) contrastive loss, while our proposed method is a TD version of the infoNCE (many-class) contrastive loss. We have clarified this difference in the final paragraph of Sec. 3.1. Empirically, TD InfoNCE is $1500 \times$ more sample efficient than its Monte Carlo counterpart ($6.5 × 10^3$ vs $10^7$ transitions) and achieves a comparable loss with $130\times$ less data than C-Learning ($7.7 × 10^4$ vs $10^7$ transitions) (See Fig. 3 (Right)). Together with the discussion below, does this address the reviewer’s concerns?

It's not clear to me what's the goal distribution is used? Is it a random goal? Does the different goal distribution affect data efficiency?

We assume that the goal space is the same as the state space and do not constraint the distribution of goal. In practice, we usually define goals using human preference. For example, in task \text{pick \\& place}, we sample the target position of the robot arm to be either in the air or on the table with the block in the gripper.

How is s_{t+} sampled? Is it the same as previous approaches - sample t from a geometric distribution?

Our TD InfoNCE is an off-policy algorithm, indicating that it does not require sample future states from an on-policy geometric distribution. Instead, we sample $s_{t+}$ randomly from the replay buffer or the offline dataset. See Appendix D.1 and Algorithm 1 for details.

The notation in Figure 1 is not very clear to me. Is it suppose to visualize Eq. 4?

Thanks for the suggestion! Figure 1 is supposed to visualize a sample-based version of Eq. 4 and our theoretical analysis in Appendix A also indicates that TD InfoNCE builds upon a Bellman equation similar to the one in this figure. The color of the edges should correspond to different terms in Eq. 4. We have updated Eq. 4 and Fig. 1 to make the notations consistent.

In Eq. 7 and the one above, should it be $p^{\pi}(s_{t+}^{(1)} \mid s’, a’, g)$ instead of $p^{\pi}(s_{t+}^{(1)} | s’, a’)$?

Thanks for finding the typo. We have updated the notation in Eq. 8.

In Table 1, the result for contrastive RL on large-diverse-v2 should not be bold; result for TDInfoNCE on unmaze-v2 should not be bold.

Thanks for the suggestion. In Table 1, we bold the success rates within one standard deviation of the best methods. In this case, the result for contrastive RL on $`large-diverse-v2`$ should be bold and the result for TD InfoNCE on $`unmaze-v2`$ should not be bold. We have revised the table.

We thank the reviewer for the detailed response and for the suggestions for improving the paper. To address the concerns about the experiments, we have run additional random seeds and clarified that the baselines mostly use the same number of parameters and batch size as our method. We have also revised the paper to address concerns about clarity. Together with the discussion for other questions below, does this fully address the reviewer’s concerns about the paper?

This doesn’t include the issue with not running hyperparameter studies on the methods independently.

As suggested by the reviewer, we ran additional experiments to study the effect of different hyperparameters for TD InfoNCE in table 2. For each hyperparameter, we selected a set of different values and conducted experiments on $`push (state)`$ and $`slide (state)`$, one easy task and one challenging task, for five random seeds. We report mean and standard deviation of success rates in Appendix E.1. These results suggest that while some values of the hyperparameter have similar effect, e.g. actor learning rate = $5 \times 10^{-5}$ vs $1 \times 10^{-4}$, our choice of combination is optimal for TD InfoNCE.

This suggests the model you are using may have more parameters than your counterparts. Is this true?

For fair comparisons, we also increased the size of neural networks in baselines to the same number whenever possible in the original submission. We have added a sentence in Appendix D.5 to clarify. We list the number of parameters for each method evaluated on online GCRL benchmarks in the table below.

As shown in the table, TD InfoNCE mostly uses the number of parameters less than or equal to baselines. Note that QRL contains a feedforward encoder $f$, a residual latent transition $T$, and a projector $d_{\theta}$ in its critic (Appendix C.3 of [2]). Even if we set the size of $f$ and $T$ to be the same as the networks in TD InfoNCE, the number of parameters in QRL is still larger than that of TD InfoNCE.

Also did you use a larger batch size for all baselines or just your algorithm?

All baselines used the same batch size as our method: 256. We have added this detail to Appendix D.5.

3 seeds is too few to get any statistical confidence …

As suggested, we have repeated experiments in Fig. 2 (Appendix Fig. 6) and Table 1 with two more random seeds (five in total) to increase the statistical confidence of our experiments. We report results on the Fetch robotics benchmark and D4RL benchmark in the revised paper. These results suggest that TD InfoNCE is stable across different random seeds and our conclusions in Sec. 4.1 and Sec. 4.2 remain the same.

This is not true as Monte Carlo estimates can be made with off-policy corrections (using Importance Sampling like in TD).

Thanks for catching this. We agree that Monte Carlo estimates can be used in an off-policy manner by applying importance weights to correct actions. We have revised the text in the final paragraph of Sec. 3.1 to note that MC methods with importance weighting could in theory perform off-policy evaluation (specifically, off-policy contrastive RL in our problem), while citing prior work that shows that off-policy evaluation based on importance weights tends to be unstable [5, 6].

… the language on motivating why a TD version of InfoNCE is oddly phrased.

We have revised this sentence in the introduction.

In your expectations above equation 7, I’m not sure what s’, a’ are here. Are you using these instead of s_{t+1} as used in equation 4? This notation should be unified.

$s’$ is the next state sampled from the environment while $a’$ is the corresponding action sampled from the goal-conditioned policy given $s’$. We have revised equation 4 to make the notation consistent.

Equation (1) and following uses, I’m not sure you explain what the superscript is signifying. I think it is time, but it is not clear from the writing.

In equation 1, we sample one "positive" $y$ (denoted $y^{(1)}$) and $N - 1$ "negative" $y$s (denoted $y^{(2:N)} = \\{y^{(2)}, \cdots, y^{(N)} \\}$). We follow the notational convention of prior work [1, 3, 4]. We have clarified this detail in the sentence before equation 1.

Thank you for the detailed response! I think you have done a great job addressing my concerns! I think you have clarified my questions and concerns. While the empirical section is still not up to the standards I would hope, this is more inline with the literature now.

Great job!

We thank the reviewer for the responses and suggestions for improving the paper. The reviewer brought up two questions regarding the experiments: selection of hyperparameters and explanation of an observation on challenging image-based tasks. We have attempted to address the question about hyperparameter selection by running ablation experiments, and we answer the question about the performance of TD InfoNCE below. Together with the discussion below, does this fully address the reviewer’s questions?

Can you explain how you selected the hyperparameters for the proposed algorithm?

We ran ablation experiments to find the best hyperparameters for TD InfoNCE (new Appendix E.1). Since our algorithm shares the same set of hyperparameters with contrastive RL, we started with the default hyperparameters from this prior work. Among those hyperparameters we chose to sweep over actor learning rate, critic learning rate, representation normalization, MLP hidden layer sizes, and representation dimensions in a range of different values (see new Appendix Fig. 10). These results suggest that while some values of the hyperparameter have similar effect, e.g. actor learning rate = $5 \times 10^{-5}$ vs $1 \times 10^{-4}$, our choice of combination is optimal for TD InfoNCE.

The γ hyperparameter is used to weight these two terms, but the choice of γ and its impact on the algorithm's performance are not discussed in detail.

The $\gamma \in (0, 1]$ is the discount factor of the underlying Markov decision process. We ran ablation experiments with $\gamma \in \\{0.90, 0.95, 0.99\\}$. Results in Appendix Fig. 10 suggests that a larger discount factor achieves a higher success rate, which can be explained by focusing more on (higher weight) correctly predicting which states are likely future states (Eq. 10).

Can you provide more details about the observation that TD InfoNCE learns on image-based pick & place and slide, while baselines fail to make any progress?

The image-based `pick \\& place` and $`slide`$ tasks are the most challenging tasks we consider – not only do they involve manipulating other objects, but also require learning such behavior from high-dimensional sensory input (i.e., images). We believe that TD InfoNCE outperforms prior methods on this task because it is able to make better use of limited data (e.g., through the temporal difference updates).

The paper focuses on fairly trivial environments, it would be nice to see these methods working on more challenging higher dimensional goal conditioned RL tasks …

Our current set of experiments on 14 tasks (8 on the Fetch robotics benchmark, 6 on the D4RL benchmark) includes tasks with 12288-dimensional (64 x 64 x 3) image observations. Prior methods struggle on at least some of these tasks (pick & place (image) and slide (image)), suggesting that these tasks are at least challenging for prior methods. While this breadth of tasks at least matches that of prior work [1, 2], we would be happy to investigate additional challenging tasks if suggested by the reviewer.

QRL assumes deterministic dynamics of the environment, while TD InfoNCE learns without such assumption.

We believe this is a strength of TD InfoNCE, rather than a weakness. In general, making assumptions about the dynamic of the environment narrows the domain where an RL algorithm might be applicable. Fig. 2b shows that TD InfoNCE can solve some tasks with stochastic dynamics while QRL suffers from a significant drop in performance compared to its result on the deterministic version. We suspect that TD InfoNCE can be applied to a wider class of goal-conditioned RL problems.

[1] Eysenbach, B., Zhang, T., Levine, S. and Salakhutdinov, R.R., 2022. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35, pp.35603-35620.

[2] Wang, T., Torralba, A., Isola, P. and Zhang, A., 2023. Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning. arXiv preprint arXiv:2304.01203.