$\pi$2vec: Policy Representation with Successor Features

Dear reviewer Kqe9, thanks for your constructive suggestions. We report our answer about the relevance of this setting and a discussion on the metrics in our general comment, and we answer the remaining questions below.

Ranking metrics in the experiments: We add additional details about metrics in our general comment and update the Appendix accordingly. To answer the reviewer’s specific question, Correlation is a ranking metric that evaluates the ranking of the estimated policies’ values, which we clarify in the manuscript. It is commonly adopted for Offline Policy Selection [1] and can be used for comparing policies. We report correlation for all our experiments to assess $\pi$2vec’s performances on ranking policies.

Visual representations: Commonly available foundation models are usually trained on visual (e.g., VIT, TAP) and text-vision data (e.g., CLIP). Those features are commonly adopted in robotics tasks [2]. Motivated by the fact that learning RL policies from vision is hard for such a task at hand, we explore how $\pi$2vec for vision-based RL benefits from visual foundation models thanks to their geometrical (TAP), semantical (CLIP), and classification-related (VIT) features. Extending $\pi$2vec for state-based representations requires foundation models that include proprioception or other state-based information as part of their input. Although state-based features are relevant for future work, in this paper, we focused on showing that visual representations are sufficient for policy representations in the domains that we considered.

Evaluation on standard offline benchmarks: We are interested in vision-based robotic environments, and we leave further evaluation of $\pi$2vec on different scenarios as future work. [3] is limited in scope to synthetic environments with a focus on state-based RL (e.g., CartPole, Walker), while [4] evaluates a broader range of environments, including synthetic (Maze2D), navigation (CARLA, AntMaze), and robotic manipulation (Kitchen) environments. In our experiments, we included the Kitchen environment along with other real and synthetic environments for vision-based robotics (Metaworld, RGBStacking, Insert gear sim and real)

[1] Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off-policy evaluation. arXiv preprint arXiv:2103.16596, 2021

[2] Mohit Sharma, Claudio Fantacci, Yuxiang Zhou, Skanda Koppula, Nicolas Heess, Jon Scholz, and Yusuf Aytar. Lossless adaptation of pretrained vision models for robotic manipulation. ICLR 2023.

[3] Yarats, Denis, et al. "Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning." arXiv preprint arXiv:2201.13425 (2022).

[4] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).

Dear reviewer eE8e, thank you for the comments and suggestions to improve our paper. We address the question about online representation in the general comment. We proceed to answer the remaining concerns below.

Limit analysis on why combining successor features and foundation models is effective: $\pi$2vec is theoretically justified by the successor feature framework [1]. We based the method on the assumption that visual foundation models consistently represent pertinent features of the environment and are adaptable as reward predictors[2]. Our focus is, therefore, on the empirical evaluation of $\pi$2vec’s representations. We validate this assumption in paragraph (iii) of the experimental Section when we validate $\pi$2vec for fully offline policy evaluation. In this setting, we fit the reward predictor $\hat{r}=\langle\phi(s), w_\text{task}\rangle$ from offline data, where $\phi$ is a foundation model, and we estimate $V^\pi(s)$ by injecting $w_\text{task}$ in Equation 1 of the main manuscript. Even if this assumption might not be true in general, we show in Table 3 that it is reasonable for the settings we consider. Table 3 reports that estimating value functions with $\pi$2vec representations is a strong offline policy evaluation method compared to FQE, a state-of-the-art method for OPE.

Comparing feature representations for $\pi$2vec: We clarify our insights about $\pi$2vec representations. We show that $\pi$2vec with foundation models always outperforms random features (see Tab. 1 and 2). We compare geometrical (TAP) and semantical (CLIP) features in Table 1 for the task Insert Gear (sim). We show that geometrical representations improve for correlation, while semantical task helps for Regret@1. Furthermore, we discuss our insight by comparing CLIP and VIT for policy representations in paragraph (iv) of the Appendix. From our analysis, we suggest that $\pi$2vec with CLIP features performs better when the historical policies do not solve tasks effectively. This is often the case when starting working on a new robotic task, e.g., insert gear (real). On the other hand, VIT features help find the best policy among a set of well-performing policies, which is ideal in ablating policies that achieve good performances for the task.

Dataset quality: we explore dataset quality in paragraph (v) in Section 7.4 of the Appendix. We collect an enhanced dataset of trajectories for all Metaworld tasks and point-of-views (12 settings). A dataset of trajectories better resembles the behavior of historical policies that are used for $\pi$2vec’s representations. We report results in Table 5, showing that $\pi$2vec improves performance when adopting an improved dataset. Similar findings hold when adopting $\pi$2vec on online data. Other aggregation methods: we thank the reviewer for the suggestion. In this initial work, we focus on proposing $\pi$2vec’s representations, and we leave aggregation strategies as further work.

[1] Barreto, André, et al. "Successor features for transfer in reinforcement learning." Advances in neural information processing systems 30 (2017).

[2] Du, Yuqing, et al. "Vision-language models as success detectors." CoLA (2023).

Dear reviewer 4YPE, thank you for the comments and suggestions to improve our paper. We report our answer about when $\pi$2vec is effective by motivating the use cases of $\pi$2vec in the general comment, and we address the remaining concerns below.

Comment on [1]: Irpan et al. [1] propose an OPE approach for robotics. Authors assume that reward is sparse and binary (success/failure in an episode), and they cast OPE as a binary classification problem. Moreover, their assumption about sparse and binary rewards does not hold in our setting, as the environments that we adopted have continuous and dense rewards. In experiment (iii) of the main paper, we compare our method to FQE, which is a strong baseline in OPE (similar to [1] but without the assumption of binary rewards), showing promising results. Coverage of historical policies: While it is true that very differently performing policies are unlikely to provide useful information for policy evaluation of the new policies, we have some experiments that look at the question of how the difference in the policies affects the performance of the method. We explore the point raised by the reviewer in paragraph (vi) in the Appendix. We adopt $\pi$2vec for representing intermediate checkpoints for a set of policies for Metaworld assembly task, and we test on held-out policies. Intermediate checkpoints for the same policy are similar to each other and different from checkpoints of other policies, especially at the beginning of training [2]. $\pi$2vec greatly outperforms actions in this setting for all metrics, proving its robustness to policy coverage.

Canonical state coverage: We sample canonical states uniformly from the dataset that was used for training offline RL policies. Even though selecting canonical states to represent the environment better is intuitively beneficial, even simple sampling at random worked well in our experiments. We conduct further experiments to ablate the state coverage. We adopt demonstrations from Metaworld Assembly task and adopt the initial state of each trajectory for $\pi$2vec’s and Action’s representation. By using the initial state of a trajectory, $\pi$2vec cannot rely on the state coverage. We report the results in the following table. We show that $\pi$2vec is robust to state coverage, showing SOTA performances even when only initial states are used. We add this experiment in the Appendix of the revised manuscript.

Assembly Left

Assembly Right

Assembly Top

Interpretation for $\Phi_\pi$: As mentioned by reviewer 4YPE, $\Phi_\pi$ represents policy $\pi$ by aggregating the successor feature representation for policy $\pi$ over a canonical set of states. $\pi$2vec performances are influenced by (i) historical policy coverage, (ii) coverage of the training data, and (iii) canonical states. Our experiments show that (i) $\pi$2vec is robust to historical policy coverage (paragraph vi and x, Section 7.4 in the revised Appendix), (ii) including online trajectories in the training set improves $\pi$2vec’s representations (paragraph v, section 7.4 in Appendix), and (iii) canonical state coverage has a limited impact on $\pi$2vec (paragraph ix in the revised manuscript in the Appendix). We hope our answers address the question and are open for further discussion.

[1] Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-policy classification. Advances in Neural Information Processing Systems, 32, 2019.

[2] Konyushova, Ksenia, et al. "Active offline policy selection." Advances in Neural Information Processing Systems 34 (2021): 24631-24644.

Thanks for the detailed response from the authors.

canonical state coverage has a limited impact

I didn't really understand the intuition behind this. To do a thought experiment, if we only hav one state, how could this approach work? I am a bit skeptical on this conclusion.

even simple sampling at random worked well in our experiments.

I am not sure if using different selection approach to approve the state coverage is a valid reasoning process. it would be great if we know how many unique states are covered, and what is percentage of those canonical states/full state, more importantly canonical states/full state sampled from high quality policy I might be missing sth but I couldn't find such info in the paper and author response.

Dear reviewer 81Fm, we appreciate the insightful feedback and reference to missing citations, which helped improve our manuscript. We address the question about online policy representation and metrics in the general comment, and next, we address the remaining concerns below.

Better analysis of the features: we present our intuition regarding the features to adopt in paragraph (iv) in the Appendix. We compare CLIP and VIT across all environments. Tables 1 and 2 show a correlation between the choice of the feature encoder and policies’ true return values. CLIP performs better than VIT in Insert Gear (Sim), Insert Gear (Real), and Metaworld. Table 4 reports policy performances in these environments, showing they are the hardest to solve, as the standard deviation among policies is high. These results demonstrate that CLIP is robust when the task is hard, and most of the policies fail to solve it. We stress that $\pi$2vec might be especially useful for this use case, as corresponds to a scenario where there is an ongoing research effort to train policies for a particular task on a real robotic platform. On the other hand, VIT is the best option when the standard deviation of policy returns are low or negligible (e.g., Kitchen and RGB stacking), making this feature most suited for ranking policies that consistently solve the task.

Baseline comparison: We point out that the action representations are extracted by running the historical policies on the set of canonical states to compute their actions. We adopt the same set of canonical states for $\pi$2vec and action representations. In this regard, the baseline and $\pi$2vec have access to the same data, and the baseline uses state information implicitly by deciding which actions should be compared. Active Offline Policy Selection (AOPS) [1] stands alone as a notable work that delves into policy representation from offline data with the task of deciding which policies should be evaluated in priority to gain the most information about the system. We argue that this is a strong baseline, as demonstrated in [1] "actions baseline" proves to be insightful for understanding policy behavior (Figure 3 in [1]) and predicting policy performance (Figure 5 in [1]). We are not aware of any previous work that addresses this particular problem by bringing foundation model representations into offline policy representation and, in particular, any other prior method that would use states in a more explicit way.

Finding the best policy: when reporting average results across tasks–Metaworld and Kitchen–we cross-validate $\pi$2vec with various features on the historical policies for each task. Next, for each task, we pick $\pi$2vec with the feature representation that achieves the lowest regret. We report the average of the per-task best representations as Best-* in Table 2.

[1] Konyushova, Ksenia, et al. "Active offline policy selection." Advances in Neural Information Processing Systems 34 (2021): 24631-24644.

Metrics ( 81Fm and Kqe9):

We adopt three standard metrics that are commonly used for Offline Policy Evaluation [1]. We report the equations below and update the Appendix of the manuscript accordingly. Normalized Mean Absolute Error: This metric is defined as the difference between the value and estimated value of a policy:

$

\textrm{NMAE} = | V^\pi - \hat{V}^\pi |

$

Where $V^\pi$ is the true value of the policy, and $\hat{V}^\pi$ is the estimated value of the policy.

Regret@1 is the difference between the value of the best policy in the entire set and the value of the best predicted policy. It is defined as:

$

\textrm{Regret @ k} = \max_{i \in 1:N} V^\pi_{i} - \max_{j \in (1:N)} V^\pi_{j}.

$

Rank Correlation (also Spearman's $\rho$) measures the correlation between the estimated rankings of the policies’ value estimates and their true values. It can be written as:

$

\textrm{RankCorr} = \frac{\textrm{Cov}(V^\pi_{1:N}, \hat{V}^\pi_{1:N})}{ \sigma(V^\pi_{1:N}) \sigma(\hat{V}^\pi_{1:N})}

$

When is each of the metrics applicable? (81Fm)

Correlation and regret@1 are the most relevant metrics for evaluating $\pi$2vec on Offline Policy Selection (OPS), where the focus is on ranking policies based on values and selecting the best policy [1]. Regret@1 is commonly adopted in assessing performances for Offline Policy Selection, as it directly measures how far off the best-estimated policy is to the actual best policy. Correlation is relevant for measuring how the method ranks policies by their expected return value. By relying on methods that achieve higher correlation and thus are consistent in estimating policy values, researchers and practitioners can prioritize more promising policies for online evaluation. On the other hand, NMAE refers to the accuracy of the estimated return value. NMAE is especially significant when aiming at estimating the true value of a policy and is suited for Offline Policy Evaluation (OPE), where we are interested in knowing the values of each policy. We assess $\pi$2vec’s representation in both settings, showing that $\pi$2vec consistently outperforms the baseline in both metrics. We improve the discussion on metrics in the Appendix of the manuscript.

[1] Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off-policy evaluation. arXiv preprint arXiv:2103.16596, 2021