We thank the reviewer for the thoughtful reviews. We appreciate that the reviewer thinks that our paper is well-written and our method is simple yet effective. Below we try to address the concerns from the reviewer. If we are able to resolve some concerns, we hope that the reviewer will be willing to raise the score.

Concern 1: evaluations restricted to deterministic environments

We agree that our evaluations are conducted in deterministic environments. We respectfully argue that most offline IL/offline RL algorithms are evaluated on deterministic environments, and the dynamics in many scenarios (e.g., robotic manipulation) are completely deterministic. Meanwhile, we do not claim that our method is all-around and can handle all scenarios (it is difficult for one single algorithm to do so). With only one expert trajectory, SEABO may fail to produce satisfying rewards for learning in stochastic environments. However, we do not think the potential incapability of handling stochastic environments downgrades the contribution of our work.

We agree that, intuitively, there may exist an inherent correlation between the number of expert trajectories needed and the stochasticity of the environment. We deem that it is interesting to check whether the stochasticity of environments affects the performance of SEABO. Nevertheless, as far as we can tell, there are few standard, widely adopted benchmarks with stochastic continuous control environments. Some prior work (e.g., stochastic D4RL Mujoco suite in Appendix C.2 in [1]) introduce stochasticity to the environment by modifying the reward in the dataset, which is somewhat incompatible with SEABO since there are no rewards in the dataset in our setting. We then introduce stochasticity by injecting Gaussian noises (with mean 0 and std $\sigma$) to observations during evaluation, following prior work [2]. We try std=0.05 and std=0.1, respectively, and conduct experiments on MuJoCo datasets. We use IQL as the base algorithm and follow the same hyperparameter setup of SEABO in MuJoCo tasks. We summarize the results below.

Table 1. Experimental results in stochastic environments. Oracle means that raw rewards are used for training. The results are averaged over 5 runs.

Based on the results, we find that under stochastic environments, the base algorithm itself fails to behave well (we observe performance collapse on many datasets like halfcheetah-medium-expert-v2). The performance of both IQL and IQL+SEABO decreases with larger noise std. Note that SEABO still outperforms vanilla rewards on many datasets under stochastic environments.

[1] Ma, Y., Jayaraman, D., Bastani, O. Conservative offline distributional reinforcement learning.

[2] Guo, Y., Oh, J., Singh, S., Lee, H. Generative adversarial self-imitation learning.

Concern 2: the paper lacks theoretical justifications

We would like to clarify that the main contribution of this paper lies in the empirical side. Many previous offline IL papers like OTR [3] also do not include theoretical analysis. Our concern lies in the fact that theoretical analysis often relies heavily on assumptions that may be hard to satisfy in practice, resulting in a gap between theory and empirical results.

[3] Luo, Y., Jiang, Z., Cohen, S., Grefenstette, E., Deisenroth, M. P. Optimal Transport for Offline Imitation Learning.

We thank the reviewer for the constructive comments. We provide point-to-point clarification to the concerns of the reviewer. Please check our responses below.

Concern 1: there are only empirical analysis and there are certain tasks where SEABO would perform poorly

We have to admit that our paper mainly lies on the empirical side, mainly due to the fact that theoretical analysis often incurs a gap between theory and practical implementation or empirical results, and some assumptions made for the benefit of theoretical derivation may not necessarily hold in practice. We agree that a theoretical analysis can be a nice addition to our work, and can leave that for future exploration.

It is interesting to explore how our method behaves when there is a precise boundary between what is acceptable and what is a failure. To that end, we use the risky pointmass environment from [1] (please see the environment code in https://github.com/JasonMa2016/CODAC/blob/main/env/risky_pointmass.py and the details of risky pointmass environment in Appendix C.1 of the CODAC paper) and make some slight modifications. In the original pointmass environment, the agent gets a penalty with some probability. To make the boundary more precise, we directly assign the agent a penalty if it steps into the risky region (line 107-111). That is,

if not self.is_safe(self.state):
    u = np.random.uniform(0, 1)
    if u > self.risk_prob:
        cost = 1
        reward -= self.risk_penalty

is modified to

if not self.is_safe(self.state):
    cost = 1
    reward -= self.risk_penalty

Other codes are kept unchanged. We collect the offline dataset by training an SAC agent for 1M interactions. We choose the trajectory with the highest return as the expert trajectory. We train an IQL agent upon the collected dataset, and set the reward scale $\alpha=1$ and coefficient $\beta=0.5$ for SEABO. As the environment is simple, we train the IQL agent for 500K gradient steps (hyperparameters are set to be identical to MuJoCo tasks) and summarize the experimental results below. In the risky pointmass environment, some of the data in the expert trajectory can be close to the risky region, while it turns out that IQL+SEABO still beats IQL with vanilla reward.

Table 1. Results of SEABO in risky pointmass across 5 different random seeds. $\mu\_{\rm{max}}$ denotes the highest return in the dataset and $\mu\_{\rm{min}}$ denotes the smallest return in the dataset.

Nevertheless, we agree that under some extreme cases (e.g., the expert trajectory is very close to the risky boundary), our method may fail. We actually do not expect that our method can handle all possible situations (and it is hard to do so). If one is concerned with the existence of a precise boundary of failure, one can add some rules to post-process the rewards calculated by SEABO (e.g., penalize those query states that are too close to the boundary).

[1] Ma, Y., Jayaraman, D., Bastani, O. Conservative offline distributional reinforcement learning.

Concern 2: SEABO will only work in lower dimensional control environments

It is true that MuJoCo tasks are low-dimensional control tasks. However, in the main text, we also evaluate SEABO on Adroit tasks, which controls a 24-DoF simulated Shadow Hand robot to accomplish different tasks like hammering a nail. The observation dimension of pen-human-v0 gives 45, and the observation dimension of hammer-human-v0 gives 46. We further include experiments on kitchen datasets, whose observation space gives 60 (which we think can be regarded as high dimensional control tasks), and the results are shown below. Based on the results in Adroit tasks (Table 3 in the paper) and kitchen tasks, we believe SEABO can also work in high dimensional control environments (the curse of dimensionality does not seem to be an issue here).

Table 2. Results on kitchen environments across 5 different random seeds.

However, it may be challenging to apply SEABO on pixel-based tasks (which are of high dimension and SEABO may suffer from the curse of dimensionality), which we leave as future work since we mainly focus on state-based tasks in this paper. As we stated in Appendix E, one possible solution is to leverage a pre-trained encoder to convert the high-dimensional input into low-dimensional representations. Then, we can apply SEABO to acquire rewards.

We thank the reviewer for the insightful review. We appreciate that the reviewer thinks that our empirical validation is thorough and the performance of SEABO is strong. Please find our clarifications to the concerns below.

Concern 1: evaluation on a challenging, long-horizon manipulation task to test the limits of the method

We thank the reviewer for recommending the IKEA Furniture assembly benchmark [1]. We think it is interesting to check how well our method behaves under manipulation tasks that require high precision. We adopt the skill-chaining [2] repository (https://github.com/clvrai/skill-chaining) from the IKEA benchmark author for running experiments, which contains the furniture benchmark. We implement the IQL algorithm on top of sac_agent.py in the robot_learning repository. We set the expectile=0.7, and tempreature=3.0 in IQL (default values for MuJoCo tasks). We set control_type=ik for demonstrate generation and RL training. We use the provided furniture_sawyer_gen.py script to generate 200 demonstrations. Though this benchmark supports IL algorithms, it seems to be prepared for online IL algorithms. Since no offline datasets are available, we train an SAC agent online for 1M steps and store its collected trajectories. We acquire the offline dataset by combining the demonstration data and the online interaction data. We randomly choose one demonstration from the generated demonstrations as the expert demonstration. We train the IQL agent offline for 1M gradient steps. Due to the fact that the whole procedure is time-consuming (collect online data, generate demonstration, train offline RL algorithm, evaluate offline policy), we only conduct experiments on two tasks from the IKEA benchmark, chair_ingolf_0650 and three_blocks. We set --num_connects=4, --max_episode_steps=800 for chair_ingolf_0650 and --num_connects=2, --max_episode_steps=400 for three_blocks. For SEABO, we set reward scale $\alpha=1.0$, coefficient $\beta=0.5$ for these tasks. We summarize the results below (as we are not familiar with this benchmark, we report some results based on the test logging information). We take the results of BC from [1].

Table 1. Results in IKEA furniture benchmark over 5 runs.

[1] Lee, Y., Hu, E. S., Lim, J. J. IKEA furniture assembly environment for long-horizon complex manipulation tasks.

[2] Lee, Y., Lim, J. J., Anandkumar, A., Zhu, Y. Adversarial skill chaining for long-horizon robot manipulation via terminal state regularization.

Note that this benchmark is quite challenging, and requires very high precision manipulation. It turns out that behavior cloning methods generally fail (please see Table 1 in [1]), and IQL with vanilla rewards also fails, indicating that the base algorithm counts in this benchmark. Online methods like skill-chaining [2] can handle these tasks well, while it remains unclear whether offline RL algorithms can achieve meaningful performance in this benchmark, which we believe lie out of the scope of our paper.

We further examine how SEABO behaves on some less challenging long-horizon manipulation tasks. To that end, we evaluate SEABO in Kitchen datasets from the D4RL paper. The kitchen environment consists of a 9 DoF Franka robot interacting with a kitchen scene that includes an openable microwave, four turnable oven burners, an oven light switch, a freely movable kettle, two hinged cabinets, and a sliding cabinet door. In kitchen, the robot may need to manipulate different components, e.g., it may need to open the microwave, move the kettle, turn on the light, and slide open the cabinet (precision is required). Please see details of the kitchen environment in [3,4].

[3] Gupta, A., Kumar, V., Lynch, C., Levine, S., Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.

[4] Fu, J., Kumar, A., Nachum, O., Tucker, G., Levine, S. D4rl: Datasets for deep data-driven reinforcement learning.

We thank the reviewer for acknowledging that our method is simple yet effective. Below we try to address the concerns of the reviewer. If we are able to resolve some concerns, we hope that the reviewer will be willing to raise the score.

Concern 1: on the use of Euclidean distance

Thanks for the advice. Section 3.2 Reward Engineering paragraph in [1] lists several works that leverage Euclidean distance for reward engineering, which we think can be a nice reference here. We have included this paper in our revised manuscript (Section 4) for validating the choice of Euclidean distance. The reviewer is concerned that there may exist some scenarios where states that are close in Euclidean distance are, in fact, far apart when accounting for the transitions within the MDP. Based on our experiments, we do not find such phenomena (Euclidean distance can incur good results on most of the evaluated tasks). Suppose that there exist some cases where such a phenomenon occurs, we would suggest the user adopt another distance measurement that can better measure their deviation within the MDP.

[1] Torabi, F., Warnell, G., Stone, P. Recent advances in imitation learning from observation

It is interesting to check how SEABO behaves with other distance measurements. We choose Manhattan distance ($D(x,y) = \sum\_{i}|x\_i - y\_i|$) and cosine distance ($D(x,y) = \dfrac{x\cdot y}{\||x\|| \||y\||}$) for evaluation and summarize the results below. Note that it is quite easy to replace Euclidean distance with other distance measurements (the modification is minor as we only need to replace the function that calculates Euclidean distance with another distance measurement function). Note that we keep hyperparameters of SEABO and IQL unchanged on the evaluated datasets across different distance measurements.

Table 1. Comparison of SEABO under different distance measurements. The results are averaged over 5 different random seeds.

It can be seen that Manhattan distance can also incur good performance on some datasets, while cosine distance seems to have unsatisfying performance on numerous datasets. The optimal distance measurement for different dataset may vary, while simply using Euclidean distance can already ensure a good performance.

Concern 2: (oracle) should be omitted from Table 1-3 and 6

By specifying oracle, we would like to emphasize that the results are obtained by running the offline RL algorithms on datasets with ground-truth rewards. We can remove these if the reviewer deems it necessary.

Dear AC, thanks for your comments. We appreciate it that you recommend [1]. We have checked this paper and would respectfully argue that SEABO is different from [1].

On the motivation

If we understand it correctly, the motivation of acquiring intrinsic reward in [1] is to check whether the state-action pair comes from expert dataset, i.e., $R=\\{(s,a)\in D\\}$. The theoretical analysis is constructed based on such reward formula. Its practical form gives $R_{int}=1-\min_{(s^\prime,a^\prime)\in D}d_{l2}((s,a), (s^\prime,a^\prime))^2$, which incurs some gap from its theory. The authors write that such relaxation can be seen as an upper bound on the scaled theoretical reward and $\min_{(s^\prime,a^\prime)\in D}d_{l2}((s,a), (s^\prime,a^\prime))^2$ denotes the $L_2$-diameter of the state-action space.

However, our motivation in this paper is quite straightforward, that we would like to determine the optimality of the single transition (instead of examining whether the transition comes from the expert trajectory or performing relaxation to the rewards). We assume that the transition is near-optimal if it lies close to the expert trajectory. Hence, we assign larger rewards to the transition if it is close to the expert trajectory and a smaller reward otherwise.

On the method

There are also many differences between the two methods,

• [1] measures the diameter of state-action space, which gives $\min_{(s^\prime,a^\prime)\in D}d_{l2}((s,a), (s^\prime,a^\prime))^2$. This is actually a special case of SEABO with Euclidean distance. However, SEABO is not restricted to Euclidean distance. Our procedure is, we first find the nearest neighbor of the query sample, and then utilize some distance measurement (different distance measurement can be used here) to decide the distance between the query sample and its nearest neighbor, and finally get the reward by adopting a squashing function. One can adopt any distance measurement to get the final reward (please see our responses to Reviewer KA4c).

• SEABO strongly relies on the nearest neighbor methods (please check our main paper), and one can use different types of nearest neighbor algorithms in SEABO, while [1] does not emphasize search algorithms. Note that different search algorithms with different hyperparameter setup can result in different final rewards.

• SEABO can also work in state-only regimes, while [1] strongly relies on the assumption that state-action pairs are present in the expert trajectory in its theory and practical implementation

On the setting

As AC comments, SEABO is targeted at the offline imitation learning setting while [1] addresses the online imitation learning setting. Meanwhile, it seems that the experimental setup of our paper and [1] also differs.

Please let us know if we miss something, or if AC is satisfied with our response. We are more than happy to update our manuscript in the next revision to include the recommended paper and discuss the differences between our work and [1]. We would upload the final revision before the end of the rebuttal.

[1] K.Ciosek Imitation Learning by Reinforcement Learning