Simple Data Sharing for Multi-Tasked Goal-Oriented Problems

1. "As the authors themselves outline in the conclusion, only AntMaze was considered. While I appreciate that the authors were open about the limited experiments, I think comprehensive experiments are something necessary to establish the strength of SDS rather than it being a “interesting future direction” As a comparison, MAHALO considers more settings on MuJoCo (D4RL datasets) as well as MetaWorld."

We chose AntMaze because it is a popular benchmark and it is easy to adapt its environments to construct offline CGO problems. (On the other hand, locomotion mujoco problems in D4RL are not goal oriented). . Besides the 3 variations, we also add goal prediction as an extra baseline and experiment with new perturbation settings (Section 4.4 and 4.5) when constructing the context-goal dataset, which helps to show the efficacy of the method in a better way.

2. "While it is true that unsupervised datasets such as task-agnostic play data can be easier to obtain than supervised datasets, SDS requires the dynamics of the dataset to cover goal-reaching trajectories. This is essentially suggesting that SDS always requires optimal trajectories, since we can just add a goal-reaching indicator at the last step of that trajectory even if it is unsupervised initially. This assumption may not scale to more complex GCRL problems if the goals are harder to reach and only near-optimal agents can achieve that goal."

We respectfully disagree with the claim that “This is essentially suggesting that SDS always requires optimal trajectories, since we can just add a goal-reaching indicator at the last step of that trajectory even if it is unsupervised initially” is not true. We’re sorry that our original manuscript might have created this misconception. We would like to clarify that our method does not directly add a goal reaching indicator at the last step since the goals context-goal only dataset might not appear in the dynamics dataset, which is the case in our newly added experiments in Section 4.4 and 4.5.

Theoretically, we only require 1) the distribution generating the dynamics dataset "covers" a feasible path to the target goal set via the generalization ability of the learned value function (which includes stitching) and 2) the context-goal distribution "covers" some feasible goal via the generalization ability of function approximation (which does not require any access to non-goal examples), which is scalable to complex settings. We also add additional explanations in our assumptions and theorems in the revision.

3. "While the theoretical results in Theorem 3.1 are nice, it is not immediately clear what the novel contributions are. "

The main technical contribution here is to show that SDS which is based on a fictitious action can be implemented with PSPI by using function approximators and their assumptions based on the original MDP without the fictitious action. The details are given in Appendix B.3 and Appendix B.4 ( Lemma B.10): We show that by carefully analyzing the Bellman equation of the action-augmented MDP, we can translate the needed assumptions back those defined in the original MDP, so we have Assumption 3.2-3.4 defined in natural terms wrt the original MDP. These techniques are novel and critically leverage the structure of CGO and SDS. The statistical analyses in Appendix B.4 (High-probability Events) are standard

Having the assumptions made in terms of the original MDP (not the augmented MDP) is important to establishing a formal theoretical result. Otherwise, the assumptions may be vacuous as they cannot be compared with others who do not introduce the augmented MDP. We will highlight the importance of these contributions better in the final draft.

4. "The introduction motivates SDS by claiming that Offline GCRL algorithms such as GoFar (Ma et, al. 2022) “can fail when the predicted goal is not reachable from the initial state.” However, this claim is also not evaluated in the experiments. Overall, there is a lack of baselines which were included in the related work section but not compared in the experiments."

Thanks for your suggestion! We also add the goal prediction baseline in Section 4.4 and 4.5.

5. "Similarly, MAHALO was not considered as a baseline for the experiments although it was compared to SDS in the theoretical results."

We did initial trials using MAHALO in the AntMaze environments, and we found that the performance is not as good as PDS+IQL or goal prediction + IQL in larger mazes. One reason may be that the public MAHALO implementation is based on ATAC and PSPI, which do not have published results and hyperparameters for Antmaze. Also, MAHALO requires adversarial training to learn a pessimistic reward function which might pose extra difficulty in the training.

Weaknesses

1. “The applied assumptions in the paper are quite strong”

As opposed to some misunderstandings of our assumptions raised by the reviewers, we highlight that we only require 1) the distribution generating the dynamics dataset "covers" a feasible path to the target goal set and 2) the context-goal distribution "covers" some feasible goals. We remark that "covering" here does not mean the such a path or a goal has to appear in the datasets or the data have to be generated only by goal reaching policies (e.g., it suffices that data can be stitched together to form a goal reaching path). Instead, it is a distribution-level coverage requirement; especially we will present coverage conditions based on the generalization ability of function approximators used in learning, which is weaker than requiring coverage conditions based on non-zero density.

2. "the experimental validation of the proposed methodology is limited"

We chose AntMaze because it is a popular benchmark and it is easy to adapt its environments to construct offline CGO problems. (On the other hand, locomotion mujoco problems in D4RL are not goal oriented). Besides the 3 variations, we also add goal prediction as an extra baseline and experiment with new perturbation settings (Section 4.4 and 4.5) when constructing the context-goal dataset, which helps to show the efficacy of the method in a better way.

Questions

1. "Is it possible to generate policies for achieving any arbitrary goal state, and if so, what are the principles for generalizing across different goals?" 2. "Given a limited set of goal contexts, what needs to be learned to derive action plans for arbitrary goals? If so, does this study propose a specific model structure or learning technique for learning such information"

1&2 are highly related so we answer them in one piece. In our CGO setting, the MDP is context-dependent, which includes goal-dependency as a special case (i.e. the traditional goal oriented problem is the case when context=goal). As opposed to learning a context dependent reward function/goal predictor in the baseline methods, we directly learn context-dependent value functions and policies via adding the fictitious transitions and fictitious actions as stated in the paper. We can generate policy for arbitrary contexts since it takes context as input, making it natural to generalize across different contexts.

5. "How can we train a reward model with only using positive data (For the usage of the reward model, can you elaborate the meaning of "choosing the percentile as 5% in the reward distribution evaluated by the context-goal set as the threshold to label goals" in Appendix C.1?"

As mentioned in 4.2, we use PDS to train a pessimistic reward function with ensembles of 10 reward functions, which measures the uncertainty of the value function prediction, making it possible to learn a pessimistic reward as introduced in PDS. Since the uncertainty term is subtracted from the reward predicting, making it a continuous reward prediction and we need to convert it back to -1/0 to label the goals, we need to select some certain threshold. So we use the tradition in outlier detection where we choose some percentile of the predicted values from the training distribution as the threshold.

Nonetheless, we also note that the pessimistic rewards learned in this way, and the more naive version RP, are not perfect and can be far away from the true rewards. Nonetheless, we saw in the experiments, IQL with these imperfect rewards can still have fairly good performance in some datasets. Such a robust behavior to reward imperfection has been recently observed and proved for offline RL in the literature [Li et al. 2023, Shin et al. 2023].

Li, Anqi, et al. "Survival Instinct in Offline Reinforcement Learning." arXiv preprint arXiv:2306.03286 (2023).

Shin, Daniel, Anca D. Dragan, and Daniel S. Brown. "Benchmarks and algorithms for offline preference-based reward learning." arXiv preprint arXiv:2301.01392 (2023).

6. "In Section 4.3, what is the reason for the observation that "for PDS, we can observe that the reward distribution for positive and negative samples are better separated in the large one than the medium one?"

For different environments, we have different context-goal dataset and trajectory only dataset. The performance of PDS relies on how well it can capture the pessimism correctly in our test context distribution given the training data. And we observe that although the medium maze is a smaller problem than the large maze, the training distribution is harder for PDS to separate the positive and negative examples for the test distribution.

7. "What are the main differences among the three experimental settings (from Section 4.3 to 4.5)? I observe a different range of context distribution but it seems that all test contexts are either fully in-distribution or partially in-distribution (Table 4) but not include real out-of-distribution cases."

The main differences are shown in Fig 1 and also discussed in 4.1: 1) the easy case where goal sets are very similar such that it could be approximately solved by a context agnostic policy, 2)We have finite contexts and the goal sets from different contexts are distinct and do not overlap 3)The contexts are continuous and randomly sampled, the goal sets from different contexts are different but could share some overlap, making a complex CGO problem. We do not aim to show out-of-distribution generalization for context distributions. However, the learned policy could enjoy some out-of-distribution generalization which would be an interesting future direction.

1. "I doubt the coexistence of separate unsupervised data and context-goal data might not be so difficult to combine… we can label a state accomplishing a given context as the goal state with ease. " (Similar: why can't we relabel the unsupervised data for a targeted task as the context can be usually derived from the state…. The authors also mention that they can use "an oracle function to tell whether a state is within the goal set".)

While what the reviewer mentioned is applicable in some special cases, such a hindsight construction is not possible for offline CGO problems in general. Here are the reasons: 1) the test context which might not even exist in the context-goal data so we cannot directly label goal states (e.g. when the context space is continuous); and 2) the exact same states in the context-goal dataset could never appear in the unsupervised dataset. Therefore, an example of this kind is our third experimental setup in Section 4.5, where the test context is randomly sampled so it could never appear in the training set (but potentially still under coverage through function approximation). In the revision, we also add new experimental results in our draft where we perturb the context-goal dataset (Section 4.4 and 4.5) such that the goals in the context-goal dataset never appear in the unsupervised dataset, and show that our method still works well. Finally, as in our footnote for the oracle function, it is only used to generate context-goal dataset for the sake of problem construction; this information is not revealed to the learning algorithms and with further perturbation of the context-goal dataset it is also not possible to “label a state accomplishing a given context as the goal state with ease”.

2. "In Section 3.1, the authors see the key problem that "some s \in G_c in the dataset is also observed in the dynamics dataset." However, I speculate that the process of Algorithm 1 cannot tackle this issue as the transition of a goal state x to x’ where will x’ \neq x^+ still remains. The problem can only be alleviated with the mentioned approach of "equally balancing the samples $\bar{D}\text{dyn} \bar{D}\text{goal}$" (in section 3.2)."

The claim that “Algorithm 1 cannot tackle this issue as the transition of a goal state x to x’ where will x’ \neq x^+ still remains” is a misunderstanding, since a^+ is not in the original action space. For example, x is a goal state, x’ is some non-goal state and there is some transition (x,a,x’) in the original dynamics dataset, and we add (x, a^+, x^+) as fictitious transitions. As a result, Q(x,a^+) would be high while Q(x,a) will be low, so V^*(x) will still be high. In general, we introduce an extra fictitious action a^+ for x to reach x^+, which is outside the original action space in the unsupervised dataset, so there is no such conflict in dynamic programming when we compute the optimal value function via argmax over all possible actions (including both real and fictitious actions). If such fictitious action and transition exists with non-zero probability, it will naturally be the highest Q value over all possible actions (e.g., via expectile regression for IQL,which does not necessarily require balancing the samples).

3. "A confusing point is that the authors provide theoretical analysis on SDS+PSPI but do not use this variant for experiments (they use SDS+IQL instead). Why not use the theoretically sound SDS+PSPI method for experiments?"

Thank you for raising this point. We choose IQL since it is a popular practical algorithm to solve AntMaze environments, making it easier to compare with the literature and other baselines. Including SDS+IQL in the experiments also shows that the idea of SDS is indeed general, not just limited to PSPI.

4. "Why do the experiments replace 0/1 rewards with -1/0 rewards? Are they intrinsically different or empirically different?"

To keep our method comparable to the context agnostic baseline, we follow the tradition to use -1/0 reward in AntMaze experiments as in the original IQL paper. The -1/0 rewards are just a scaled and shifted version of the 1/0 rewards (specifically, first scale the reward of 1/0 by $\frac{1}{1-\gamma}$ to the reward of $\frac{1}{1-\gamma}$ / 0 and then shift the reward at each state by -1; then we get the -1/0 reward). Therefore these two reward systems induce the same policy ranking, and our theory applies.

Thanks for the authors' rebuttal that addresses some of my questions. After reading the revision and other reviews, I still have concerns about the discrepancy between theories and practical solutions. Besides, I think the authors should consider changing the experiment design since current setup of different goal levels does not help show different properties of SDS and also lack diverse benchmarks. I will keep my score.

Limitations

1. “The proposed approach is only applicable to goal-conditioned tasks with rewards of 1 only at goal states and zero rewards otherwise.”

We agree such goal oriented problems do not cover the full spectrum of RL problems, but they are quite common in practice, especially given many real-world problems do not come with dense rewards naturally. That is why goal conditioned RL (GCRL) problems are widely studied in the literature. We choose the sparse reward (only 1 at goal states and 0 otherwise) setting as it is a realistic and commonly-used setting in goal conditional RL where the reward is sparse. It is studied in most of the GCRL papers [1]. We should note that this setting also covers problems generated by applying any reward shifting/scaling (e.g.,GCRL problem with -1 at each step and 0 at the goal) since they will induce the same policy ranking.

[1] Goal-conditioned reinforcement learning: Problems and solutions. M Liu, M Zhu, W Zhang

2. "The paper makes very strong assumptions about the offline dataset. Mainly, they assume that the goals dataset D_goal contains all the goals the agent will encounter and that the trajectories dataset (D_{dyn}) contains trajectories leading to those goals.”

The assumption stated here is indeed a misinterpretation of our theory. We are sorry if our original manuscript’s writing is not accurate. To clarify, we only require 1) the distribution generating the dynamics dataset "covers" a feasible path to the target goal set and 2) the context-goal distribution "covers" some feasible goals. We remark that "covering'' here does not mean that such a path or a goal has to appear in the datasets or the data have to be generated only by goal-reaching policies (e.g., it suffices that data can be stitched together to form a goal-reaching path). Instead, it is a distribution-level coverage requirement; especially we present coverage conditions based on the generalization ability of function approximators used in learning, which is weaker than requiring coverage conditions based on non-zero density. We also add additional explanations in our assumptions and theorems in the draft.

Theory:

1.“The theoretical assumptions seem imprecise or incorrect …the authors say that the state, action, and context spaces can be continuous.”

Thank you first for raising this point. As we discussed in the footnote of the theorem, we only use the discrete notation (i.e. writing the cardinality of function and policy classes) for simplicity and “We state a more general result for non-finite function classes in the appendix”. In Appendix B4, we introduce the definition of covering number and replace |\Pi| with the covering number of the function class.

2. “It would have been helpful if the authors explained step-by-step each term of the bound and how that leads to a small regret with high probability”

Certainly, we can interpret the upper bound in the theorem as follows: the statistical errors would decrease as we have more data from $\mu_{goal}$ and $\mu_{dyn}$; then with the finite coefficients, the final upper bound would also decrease. Take $\pi = \pi^*$ as an example. For the coefficients to be finite, it means 1) the state-action distribution from the dynamics data ``covers" the state-action pairs generated by $\pi^*$, which naturally includes stitching; 2) the support of $\mu_{goal}$ "covers" the goals $\pi^*$ would reach. The coverage is measured based on the generalization ability of $f$ and $g$; e.g., if $g(x_1)$ and $g(x_2)$ are similar for $x_1\neq x_2$, then $x_2$ is within the coverage so long as $x_1$ can be generated by $\mu_{goal}$.

3. “The motivation for why Algorithm 1 is expected to work is not given…In fact, the algorithm looks like it shouldn't work, since it gives zero rewards for all states in D_dyn potentially including goal states that were given rewards of 1 in D_goal. It is not clear why this reward scheme makes sense”

We present why our augmented MDP is equivalent to the original MDP in the sense of regret in Appendix B1, and add Remark 3.1 as an intuition. The core idea is that by introducing an extra “fictitious” action $a^+$, we create some fictitious transition from the potential goal states to the terminal state as future transitions after reaching the goal state. For such potential goal states, the positive future reward will be propagated back to them with a high value by Bellman equation. For example, if s0->...->g is a trajectory to some potential goal g, adding future transition g - > s+ with action a^+ and reward one, and labeling all previous trajectories with reward 0 will not affect the ranking of the learned value functions.

4. Presentation of the theoretical results

Thanks for your suggestions. We reorganized Section 3 accordingly (e.g., explanations, definitions). Please check the revised draft.

Experiments

1. “All the experiments are in a single domain (the D4RL AntMaze)”

We chose AntMaze because it is a popular benchmark and it is easy to adapt its environments to construct offline CGO problems. (On the other hand, locomotion mujoco problems in D4RL are not goal oriented). . Besides the 3 variations, we also add goal prediction as an extra baseline and experiment with new perturbation settings (Section 4.4 and 4.5) when constructing the context-goal dataset, which helps to show the efficacy of the method in a better way.

2. 5 random seeds Our empirical results are from over 100 evaluations from each random seed, and then averaged over 5 random seeds for each our empirical results are from over 100 evaluations from each random seed, and then averaged over 5 random seeds for each environment, which follows the convention of evaluation metrics in AntMaze.