Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning

Dear reviewer 6wvZ, thank you for the positive feedback, detailed comments and constructive suggestions, we have addressed the concerns and suggestions as follows.

---

Comments

C1. : Some discussion is probably missing about how the method, combined with Q-learning, is likely to propagate values from the expert-proximal states to earlier states of the same trajectory.

Response: We appreciate the reviewer for bringing this to our attention. We have responded to this by incorporating relevant information at the end of Section 3.2 in our updated manuscript, in which we elaborated on how the proposed method, in conjunction with an agent (specifically the soft actor-critic (SAC) used in this paper), propagates values from states proximal to the expert to earlier states within the same trajectory.

C2. : A missing ablation study is one that ablates out $B^-_{Reversed}$ as this seemed like an arbitrary choice when presented in Eq.9… i’d be curious to see what the performance without it looks like.

Response: We would like to bring to the reviewer’s kind attention that the ablation study presented in Table 2 of the original main manuscript, specifically in the column labeled “w/o Hard Negative Samples”. As discussed in Section 3.3, hard negative samples for the contrastive learning of the transition discriminator are denoted as $B^-{Reversed}$. These hard negative samples can consist of different combinations of data from reversed transitions, depending on prior knowledge about different environments. To maintain generality, we refer to these as hard negative samples throughout this paper, using the term to represent $B^-_{Reversed}$.

The experiments that ablate hard negative samples demonstrate that in simpler environments, the information contained in these samples may not be necessary. In contrast, their absence in the Humanoid-v3 environment impairs performance. This difference can be attributed to the vast state space of Humanoid-v3, which reduces the likelihood that easy negative samples encapsulate the essential information contained in hard negative samples. Furthermore, given the complexity of Humanoid-v3, the agent may be sensitive to incorrectly estimated rewards that stem from the absence of hard negative samples.

C3. : Since $\beta$ is set to 0 in the main experiments, there should be some ablation study on it since it is proposed in the overall reward function of the method.

Response: We appreciate the reviewer's suggestion and have augmented the ablation study on the hyperparameter $\beta$ in Table A3 of the supplementary material. The study reveals that by setting $\beta$ within the range of [0.1, 0.9] in $R_{agg} = \beta R_{IRL} + (1-\beta) R_{TDIL}$, the agent consistently achieves expert-level performance, without the use of Behavioral Cloning (BC) loss. This result validates efficacy of the proposed reward function $R_{agg}$.

C4. : The adroit experiments would also benefit from a comparison against at least another method other than BC for completeness.

Response: We appreciate the reviewer's suggestion and have incorporated an experiment shown in Fig. A3 of the supplementary material. This includes comparisons with BC, CFIL, and f-IRL. The results demonstrate that TDIL achieves expert-level performance (i.e., a 100% success rate) within one million steps and surpasses the success rate of BC(10%), f-IRL(40%), and CFIL(10%).

C5. : Figure 1(a) is referenced in the intro yet doesn’t appear until page 4. To make the intro easier to read, it should be put earlier in the paper (it’s on page 4) or separated into multiple figures so that the relevant parts of Figure 1 appear when referenced in the intro.

Response: We appreciate the kind suggestion from the reviewer. We have relocated Fig. 1 to page 2 to ensure it appears earlier in the paper for improving the alignment with references in the introduction.

C6. : There’s some color-coding going on in Figure 1(b)-(d), (f)-(h) but they’re not explained in the figure or caption (presumably gray intensity in (b)-(d) is for the reward but I have no idea what the colors for (f)-(h) mean)

Response: We appreciate the reviewer's suggestion and have incorporated enhancements in the updated manuscript.

The color-coding in Figs. 1 (b)-(d) serves to visually represent the density of each reward signal. Meanwhile, in Figs. 1 (f)-(h), the color-coding is intended to visualize the actions taken by the agent. This approach is analogous to techniques employed in optical flow research, where color is used to provide an additional representation of direction beyond the traditional arrow visualization.

We also provide a reference in Section A.2.4 of the supplementary material to illustrate the relationship between color and direction. This is intended to offer a more comprehensive understanding of the color-coded information.

Dear reviewer eL6K, thank you for the constructive comments and feedback, we have addressed your concerns as follows.

---

Comments

C1. : In many existing works, people have discussed methods that can try to encourage the agent to go back to the expert distribution such as FIST [1]. Among IRL methods, PWIL and OT [2] also incorporate a dense distance-based reward that encourages the agent to stay close to the expert per-transition. I am wondering why the proposed method is better than these prior approaches.

Response: We would like to thank the reviewer for raising these questions.

Although prior approaches such as FIST [r1], PWIL [r2], and OT [r3] adopt strategies to motivate agents to return to the expert distribution, the distance measuring functions they employ, including Euclidean and Cosine distances, are not theoretically guaranteed to achieve this objective. Furthermore, if the reward function does not consider the environment dynamics, it may mistakenly direct the agent to states that are not genuinely “close”, and may lead to the agent becoming stuck in those states. Therefore, the awareness of environmental dynamics is crucial for the reward function. Consider the toy example in Fig. 1 (a) (discussed in Section 3). When using the L2 distance (i.e., Euclidean distance) as the metric for measuring closeness, the reward system would inadvertently encourage the agent towards all adjacent states of the expert states, even if a barrier (indicated by a red line) separates the agent’s current state from the expert state. As a result, as shown in Fig. 1 (f), even after training, the cells on the inner side of the L-shaped barrier still exhibit an incorrect policy (pointing to the bottom left).

To address these issues, a key insight of our paper is the use of a transition discriminator to obtain an environment-dynamic-aware reachability metric. This metric does not rely on the L2 distance. Instead, our transition discriminator considers transition reachability and thus is capable of providing more meaningful guidance for the agent to transition to the expert support and achieve expert-level performance in various environments.

In addition, our breakthrough lies not just in introducing a more theoretically sound distance measuring function but also in densifying the single demonstration from a one-dimensional manifold to an n-dimensional support. This enhancement facilitates the alignment of the demonstration with the agent support. The significance of this lies in the fact that a demonstration restricted to a 1-manifold does not overlap with the agent support in a low-dimensional manifold, as discussed in [r4]. We have elaborated on this in the updated version of Section 3.1 in our manuscript.

[r1] Hakhamaneshi et al. Hierarchical Few-Shot Imitation with Skill Transition Models. NeurIPS 2021.

[r2] Dadashi, Robert, et al. Primal Wasserstein imitation learning. arXiv:2006.04678 (2020).

[r3] Haldar et al. Supercharging Imitation with Regularized Optimal Transport. CoRL 2022.

[r4] Arjovsky, Martin, and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv:1701.04862 (2017).

C2. : If the agent has gone off the expert distribution, the proposed reward function can still degenerate. For example, if the minimal distance from a state to any expert state is 5 steps, the proposed reward will output 0 no matter what action the policy chooses. Then, how could the proposed method guide the policy?

Response: We would like to thank the reviewer for raising this question.

We wish to draw the reviewer’s attention to the fact that for states multiple steps away from expert proximity, RL methods like the soft actor-critic (SAC) used in this work can propagate the rewards to other states. This propagation occurs through value functions and the agent's interaction with the environment via exploration and exploitation.

A key insight of our method is the densification of the single demonstration’s 1-manifold into a theoretically sound n-dimensional support. This approach enables the RL agent to reach those states during exploration.

On the other hand, while methods like PWIL provide rewards for more states, they lack theoretical soundness. This is due to their use of Euclidean (L2) or Cosine distance measurements, which do not account for environmental dynamics, as discussed in Section 3.1. In addition, while our method has already achieved optimal performance in our experiments with a 1-step setup, it retains the flexibility to extend to $n$ steps if necessary. This extension can be achieved by augmenting $(s_t, s_{t+m}), 1 \leq m \leq n$, into the positive example set $B^+$. The theoretical details are provided in Section A5 of our supplementary material.

C3. : The proposed method integrates BC in its training. However, it looks like all the baselines the authors compare do not involve BC in training. This makes the comparison unfair.

Response: We would like to clarify the question as follows.

Firstly, consider the scenario where BC loss is utilized. Table r1 presents the performance of CFIL, PWIL, f-IRL, and TDIL with BC loss. It is observed that although some baselines have improved their performance with BC loss (while others worsened), TDIL consistently outperforms all these baselines.

Second, we wish to highlight that the proposed reward function is actually $R_{agg} = \beta R_{IRL} + (1 - \beta)R_{TDIL}$ (refer to Eq. (5) in the manuscript). Using $R_{agg}$, our proposed method can train an SAC agent without employing BC. The results, as reported in Table r2 (which corresponds to Table A4 in the supplementary material), reveal that even without BC, our agent can easily reach expert performance for $\beta \in [0.1, 0.9]$.

Considering these two perspectives, it becomes evident that the proposed surrogate reward $R_{TDIL}$ can effectively assist the agent in consistently reaching expert performance. This is achievable as long as the agent is trained to perform the expert action on the demonstration since $R_{TDIL}$ only motivates the agent to move toward the expert’s proximity support and does not dictate specific actions to be performed on the expert states.

• Table r1 Performance of baselines with BC loss

• Table r2 Performance of TDIL under different $\beta$ value selection.

C4. : Sensitivity analysis. The proposed algorithm involves a hyperparameter but the authors do not discuss the sensitivity of the performance with respect to different alphas.

Response: We would like to thank the reviewer for the suggestion concerning the sensitivity of performance to the hyperparameter $\alpha$. To address this, we have included two additional results in the revised manuscript.

• Table A4 in the supplementary material presents the accuracy of transition discriminators trained with different values of $\alpha$. The results demonstrate that all setups of the transition discriminator can achieve an accuracy of over 0.988 for both positive and negative examples.
• Furthermore, Fig. A6 in the supplementary material illustrates the training curve of the agent with different choices of $\alpha$. These results indicate that the majority of the $\alpha$ settings can attain expert performance.

Drawing on the findings from these experiments, we can infer that the hyperparameter $\alpha$ is relatively straightforward to calibrate, indicating its robustness and ease of adjustment in different scenarios.

Dear 9w53, thank you for the constructive comments and feedback, we have addressed your concerns as follows.

---

Comments

C1. : What is the difference between the single demonstration and sparse/scarce expert data, which one is the source of the sparse reward issue in single-demonstration imitation learning?

Response: As discussed in Section 1 and mentioned in the other sections of the main manuscript, single demonstration imitation learning (abbreviated as single demo IL) differs from conventional IL. In conventional IL, an agent typically has access to a number of expert demonstrations, which could range from hundreds to thousands, as in the case of D4RL [r1]. In contrast, the most challenging aspect of single demo IL is the limitation to only one trajectory accessible to the learning agent as a demonstration. This challenge arises because obtaining expert demonstrations is often expensive and can incur prohibitive efforts. In such a single demo setting, the demonstration is sparse compared to conventional IL settings. Utilizing conventional inverse reinforcement learning (IRL) approaches in the single demo setting would result in a sparse reward scenario.

C2. : There is no evidence to show the relationship between the convergence speed and sparseness of the rewards. Moreover, there are no statistical results to directly show the sparseness of the IRL’s rewards.

Response: We would like to address the reviewer's question as follows. Consider a continuous environment with n dimensions. From a mathematical perspective, the reward sparseness of conventional IRL compared to TDIL can be described as follows:

• For conventional IRL, one demonstration trajectory corresponds to only a 1-manifold (i.e., a 1-dimensional line).
• In contrast, the TDIL method includes all states that are one-step reachable from the single demonstration trajectory. As a result, the reward function formulated by TDIL corresponds to an n-dimensional support and is infinitely denser than any 1-manifold [r2]. A crucial aspect is to ensure that the demonstration is a support, as it is impossible to match an agent's support with a 1-manifold. We have enhanced Section 3.1 in the updated manuscript to improve accessibility and readability.

C3. : The indicator can only output binary values for state-action pairs, which will result in sparse rewards. It needs to be clarified how such an indicator can be used to define a denser surrogate reward function. A potential explanation in section 3.2 is that the surrogate reward function encourages the agent to navigate to states that are in expert proximity. What is expert proximity?

Response: We would like to address the reviewer's question as follows.

As discussed in Section 1 and defined in Section 3, expert proximity is a set of states where each state in this set can reach an expert’s demonstration state in one step. As discussed in Section 3, TDIL provides reward to the state-action pair $(s_t, a_t)$ if $s_{t+1}$ is in expert proximity.

Please note that outputting binary values for state-action pairs does not affect the reward density. The transition discriminator only tells the one-step reachability. The sparseness or denseness of the reward system only depends on the size of the manifold that can offer non-zero rewards. We increase the density by densifying the non-zero reward from a one-dimensional manifold to an n-dimensional manifold. For example, in a high-dimensional continuous environment such as Mujoco, there will only be at most 1,000 state-action pairs provided by expert demonstrations. However, the states in expert proximity are infinite. As a result, the number of state-action pairs that can reach a state in expert proximity is also infinite. We have enhanced Section 3.2 in the updated manuscript for improved accessibility.

[r1] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).

[r2] Arjovsky, Martin, and Léon Bottou. "Towards principled methods for training generative adversarial networks." arXiv preprint arXiv:1701.04862 (2017).

Dear Reviewer Ka6u, thank you for your positive feedback and encouraging review. We have addressed your concerns as follows:

---

Comments

C1. Additional experiments on a larger domain set: Locomotion environments tend to be somewhat easier for RL agents (evidenced by a variety of self-supervised methods learning to walk). I would recommend increasing the scope of the experiments to a robotic manipulation domain which would allow readers to understand the limitations of the work.

Response: We would like to thank the reviewer for raising this question. We wish to draw the reviewer’s attention to Section A.4.3 and Fig. A3 (or Section 5.4 and Fig. A4 in the old version), where we presented the experiment in the AdroitHandDoor environment [r1]. This environment is a task of the Adroit manipulation platform, which features a Shadow Dexterous Hand attached to a free arm with up to 30 actuated degrees of freedom. The results demonstrated that the proposed TDIL method attained expert-level performance (100% success rate) within one million steps and was able to surpass the performance of BC(10% success rate), CFIL(10% success rate), and f-IRL(40% success rate).

C2. Ablations utilizing high dimensional states: How does the method perform when utilizing demonstrations from the visual domain? This is necessary for completeness as there are several recent works, for instance FISH https://arxiv.org/abs/2303.01497, RoboCLIP https://arxiv.org/pdf/2310.07899.pdf which learn a reward from visual observations from a single demonstration.

Response: Thank you for mentioning these two highly related papers. We have cited them in our latest version. We would like to address this question by elaborating on the differences in settings between these methods and ours.

In the case of RoboCLIP [r2], the reward signal was generated using a pretrained video-and-language model (VLM) that was trained with additional data. In contrast, our proposed TDIL method trains the agent from scratch.

Regarding FISH [r3], it employed cost functions such as Euclidean (L2) and Cosine distance to derive rewards. However, such cost functions are not theoretically guaranteed since they do not consider the dynamics of the environment. For example, two states might be close in Eurclidean (L2) distance, but they can not reach to each other quickly since there are barriers between them. In contrast, our reward function is specifically designed to capture these dynamics, which is a primary focus of our paper. By considering dynamics, TDIL achieves superior performance, as demonstrated in our motivational experiment (please kindly refer to Fig. 1) and in the experiments detailed in Section 5 (Note that PWIL uses L2).

In this paper, our main focus is to introduce a transition discriminator-based approach to enhance reward density and facilitate one-demonstration imitation learning scenarios. In the future, exploring the leverage of high-dimensional inputs (e.g., images) would be an interesting direction to investigate.

[r1] Rajeswaran, Aravind, et al. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. arXiv:1709.10087 (2017).
[r2] Sontakke, Sumedh A., et al. Roboclip: One Demonstration is Enough to Learn Robot Policies. arXiv:2310.07899 (2023).
[r3] Haldar, Siddhant, et al. Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations. arXiv:2303.01497 (2023).