AED: Adaptable Error Detection for Few-shot Imitation Policy

Dear Reviewer upjt,

We sincerely appreciate your insightful comments and positive assessments. We now address the questions and issues you raised.

The issues listed in the weaknesses and questions are denoted as Wx and Qx, respectively, where x is the issue's index.

What’s the use case of the proposed system if it only detects failure after the fact? (W3, Q3)

Thank you for this question.

The fundamental assumption of our work is that we are monitoring the few-shot imitation (FSI) policies, and they usually have the following settings:

• Training and testing environments are disjoint. (base env. vs. novel env.)
• No exploration before inference in novel environments.
• The expert only provides demonstrations in the novel environment but may not be around the robot when it executes.

These settings make our AED methods unable to get the information of human assistance (e.g., intervention, labeling new data) or world dynamic model as used in the references the reviewer listed. A practical scenario of our AED study is provided at the end of our response.

Moreover, we are dealing with robot manipulation tasks, where the robot interacts closely with objects. Therefore, in our definition, an error (failure) occurs (right) after a failed interaction. From this moment, the current observation is inconsistent with demonstrations.

In addition, to our knowledge, most existing studies related to failure detection on robot manipulation also detect the error after the fact, e.g., the reference [1] the reviewer listed.

We are not saying that our setting is better than the one in [3], but the settings, targeted policies, and scenarios between ours and [3] are different. Thus, in our AED task, the goal is to detect the error as early as possible (after the fact). And our PrObe detects errors at the most accurate timing.

Regarding the issue of delaying 5 frames, the camera collects the observations with fps=20. It means that our PrObe only causes a delay less than 0.25 seconds, which should be acceptable in practice. From our demo video, our PrObe can detect the error before the mug falls on the table. We believe the AED problem and proposed PrObe suit many real-world situations.

A practical scenario of our AED problem:

An intelligent robot (containing policy and AED method) is trained in the factory with daily life tasks. Then, it is sold to the customer's house to perform those tasks. The customer will provide few demonstrations for the robot but may not stay around the robot for the whole time. The robot itself should be able to recognize if it performs the task correctly or not. Also, it should terminate the policy and wait for expert's help or apply the recovery policy (in our appendix) when it detects a behavior mismatch. Note that in this case, the expert is the customer, who can provide demonstrations but may not be able to label corrected actions for the robot.

How does the method compare with recent papers, such as MoMaRT, ThrifyDAgger, etc.? (W1, Q1)

Thank you for recommending these references.

As explained in Q3, in our setting, we don’t have access to human assistance (intervention and newly labeled data) and environmental exploration before inference, which are the natural limits of the FSI literature. Therefore, the works that need human-in-the-loop (reference [2]) or the world model (reference [3]) cannot be compared with our work.

The reference [1] is already included in our references. However, as we pointed out in the Introduction, these reconstruction-based methods (one-class classification) suffer when dealing with our problem.

They monitor the policies based on the reconstruction errors of observations. However, they will raise errors for every frame in the novel environments since they did not see them before. We have tried to add [1] as one of our baselines, but we cannot find appropriate thresholds since it yields large errors no matter if the frame is normal or an error in the novel environments.

Moreover, we actually build our baselines with various attributes, including policy dependent, OCC (trained with only normal rollouts), temporal information, and demonstration-conditioned. Please find them in Appendix Table 3. Besides, there are also powerful models in our baselines. For example, DCTED (2021) is a transformer model, which should be more complicated than cVAE used in [1].

However, we are grateful for receiving the references related to our work. We think summarizing and introducing these interactive error detection works is valuable, even though our settings are different. We will include them in our final paper.

Please let us know if there is any other paper the reviewer thinks is related to us or fits our setting and should be compared.

Thank you again for suggesting the references.

Please find the rest of our response below.

Dear Reviewer BbBE,

We sincerely appreciate your insightful comments. We now address the questions and issues you raised.

The issues listed in the weaknesses and questions are denoted as Wx and Qx, respectively, where x is the issue's index.

Which parts of the framework in Figure 2 do our PrObe contribute to? (W1, Q2)

Thanks for the question.

Our PrObe is an AED method (i.e., adaptable error detector, denoted as $\phi$ in Figure 2), which involves the phases of AED training and AED inference.

The framework of our AED problem contains three phases:

• Policy training phase: The few-shot imitation policy $\pi$ first trains on the base environments.
• AED training phase: The AED methods also train on the base environments. In this phase, some methods, e.g., our PrObe, might leverage the knowledge from the trained policy.
• AED inference phase: The trained policy $\pi$ performs the FSI tasks in novel environments. AED methods aim to monitor the policy and report timely once the policy behavior error occurs, i.e., the policy rollout is inconsistent with demonstrations provided by the expert in the novel environments.

Thus, all AED methods involve the last two phases of Figure 2; however, our PrObe can tackle the challenging AED problem with superior performance. We will make this more clear in the revised version.

Besides, please note that the FSI tasks are semantically similar in the base and novel environments (e.g., both are pick&place tasks). However, the interactable objects and backgrounds are disjoint between them.

A practical scenario of our AED problem:

An intelligent robot (containing policy and AED method) is trained in the factory with daily life tasks. Then, it is sold to the customer's house to perform those tasks. The customer will provide few demonstrations for the robot but may not stay around the robot for the whole time. The robot itself should be able to recognize if it performs the task correctly or not. Also, it should terminate the policy and wait for expert's help or apply the recovery policy (in our appendix) when it detects a behavior mismatch.

What's the relationship between Eq. 1, 2 and 3? (W1, Q2)

Thanks for the question.

Eq. 1 and 2 are the formal representations of how the AED method predicts the error status and how they will be evaluated.

• Eq. 1: The AED method $\phi$ takes rollout history (observations) $h$ and few demonstrations $\mathcal{D}^{n}$ as inputs and then predicts the error status $\hat{y}$.
• Eq. 2: The AED method is evaluated by its prediction $\hat{y}$ and ground-truth $y$ across all novel environments.

Eq. 3 is a new objective we proposed to train the PrObe, which is a temporal-aware variant of triplet loss. The complete objectives of PrObe can be found in Appendix Section B.

Why do we choose these operations and probabilities in our designed rollout augmentation (RA)? (Q1)

Good question.

As mentioned in the main paper's Section 5.1, the four operations (keep, drop, copy, and swap) can inject different behaviors into the pre-collected rollouts during the AED training phase.

• Dropping consecutive frames can speed up the movements.
• Copying consecutive frames can slow down the movements.
• Swapping consecutive frames can provide unsmooth movements.

Moreover, the copy operation could be replaced by the interpolation operation. We have tested this replacement, but no significant performance difference is observed. Meanwhile, the interpolation operation is very time-consuming. Thus, we still apply the four operations above.

Regarding determining their probabilities (0.3, 0.3, 0.2, 0.2), we observe that the policy usually behaves faster than the pre-collected rollouts. Thus, we would like to have a higher chance of generating speeded-up rollouts during training.

Nevertheless, the probabilities can be easily adjusted to cover more diverse rollout behaviors, which may benefit the AED training. If the reviewer wants to try any combination of the probabilities, we would love to verify it.

However, the AED methods' performance is not sensitive to the probability setting, as we tested before.

Please let us know if our responses have addressed your questions and issues.

Besides, feel free to point out if there is any other issue or topic you are interested in or think will make our work more convincing. We look forward to having active and insightful discussions with you!

Thanks again for providing valuable comments.

Dear Reviewer Zod3,

We respectfully disagree with your assessment.

We will address the raised concerns now, and look forward to having a constructive conversation with you.

The issues listed in the weaknesses and questions are denoted as Wx and Qx, respectively, where x is the issue's index.

The problem and its challenges itself are unclear. What does "No impulse signals in behavior errors" mean? (W1)

We propose a new problem named adaptable error detection (AED), which aims to detect the behavior inconsistency between the few-shot imitation policy rollouts and demonstrations.

The proposed AED contains three challenges, which makes it more challenging than an existing video few-shot anomaly detection problem. The second challenge, “no impulse signals in behavior errors,” means no significant difference between the consecutive frames when the error happens. We also provide two cases with detailed visual explanations in Figure 1.

It is worth mentioning that all other reviewers agree our proposed AED problem is crucial for robot learning when considering the safety perspective.

Presentation Issues (W1, W5)

We are perplexed about these issues.

For W1, there are only three new abbreviations in our paper, i.e., AED for adaptable error detection, PrObe for Pattern Observer, and vFSAD for video few-shot anomaly detection.

All remaining abbreviations are well-known or the method names from existing literature. For example, OCC for one-class classification, OOD for out-of-distribution, or DC for demonstration-conditioned. Also, we usually write the full name of the abbreviation at the beginning of each section if it does not frequently appear.

For W5, we do not find any inappropriateness when emphasizing the importance of our work since we propose a new problem in this paper. Most importantly, the two cases the reviewer mentioned are not even our proposed problem. Instead, they are the two problems related to us. We state the fact that they are relatively new and have drawn the researcher's attention recently.

At last, we want to highlight that we got >= 3 presentation scores from all other reviewers. We would appreciate it if the reviewer could point out which parts of the paper are unclear so that we can further improve it.

The papers make many claims without supporting evidence, e.g. "We emphasize that addressing AED is as critical as enhancing FSI policies’ performance". (W2)

It is the motivation of our work, which we spent more than 13 lines (first two paragraphs of Introduction) to describe the situation that the few-shot imitation policies may provoke damage when applying them to real-world scenarios. Thus, we should also consider promoting their ability for safety executions rather than the performance of their own tasks (e.g., manipulation tasks).

We believe this motivation is clearly presented, which all other reviewers recognize.

We would appreciate it if the reviewer could point out other unsupported claims or motivations we made in the paper. We would love to explain or improve it.

Works related to OOD detection in time-series data should be compared. (W3)

Thank you for pointing out this question.

The fundamental assumption of our work is that we are monitoring the few-shot imitation (FSI) policies, and they usually have the following settings:

• Training and testing environments are disjoint. (base env. vs. novel env.)
• No exploration before inference in novel environments.
• The expert only provides demonstrations in the novel environment but may not be around the robot when it executes.

Thus, the traditional methods on OOD detection in time-series data are unavailable. In our case, the error detector works in an unseen environment. And we only have few demonstrations in the environment which makes it hard to compute those p-values in the reference [1] or other information in the frequency space.

Moreover, the OOD samples defined in our work is the inconsistency between the policy rollout and demonstrations, which have two time-series. The OOD samples can be only detected by comparing these two video streams rather than simply computed on any one of them.

The vFSAD problem is the most related problem we can find. But our problem raises more challenges than the vFSAD problem, as we stated in the Introduction.

Please let us know if there is any other paper the reviewer thinks is related to us or fits our setting and should be compared.

Thank you again for suggesting the reference.

Please find the rest of our response below.

As the authors requested, here are just a few issues with the writing at the start of the paper that made it hard for the reviewer to understand the paper. I've provided rewritten statements or rewritten statements phrased as questions if I was not sure how to rephrase a sentence.

adjacency frames

--> "adjacent frames"

Online detection (lacking global temporal information)

--> do you mean "Online detection has to take place without any information about the timestamp of the particular state"?

a robot solves a set of everyday missions by giving a few demonstrations from the owner

--> "a robot solves a set of everyday tasks given few demonstrations of performing these tasks". Your original sentence made it unclear if the robot provides the owner of the robot with demos or if the robot is given demos.

task misunderstanding [Figure 1]

--> I am not sure what this image even represents. Is the agent doing the opposite of the expert demonstration?

... learning in policy features also has additional knowledge related to FSI tasks

--> learning policy features provides additional knowledge

(1) AED monitors the policy in novel environments whose normal states are not seen during training. Although vFSAD also monitors unseen videos, the anomalies they monitor are objects, which in AED are policy behaviors. (2) Unlike the anomaly in vFSAD raises a notable change in the perception field (e.g., a cyclist suddenly appearing on the sidewalk (Mahadevan et al., 2010)), no impulse signals appear when behavior errors occur. Either (2-a) minor visual difference or (2-b) task misunderstanding is hard to recognize through adjacency rollout frames. (3) AED requires online detection to terminate the policy before causing disasters, lacking global temporal information

--> It is unclear how a problem (AED or vFSAD) "does" anything. These words are used without any explanation: "impulse signals", "global temporal information".

But, here is my attempt to rephrase your paragraph: "(1) In the AED problem, unlike the vFSAD problem, novel anomalies are actions performed by a policy rather than objects that appear in a video. (2) The observed changes are subtle in the AED problem unlike vFSAD. (3) The AED problem, unlike vFSAD which detects in anomalies in videos that have already been recorded, does not have any information from the future and hence is more challenging."

unvanquishable obstacles

--> challenging obstacles

... to demonstrate the practice of the proposed AED problem

--> ... to demonstrate potential next steps after the AED problem is addressed.

the community explores the paradigm for policy learning ...

--> the wider community has explored various solutions for learning from demonstrations such as ...

Notably, they either assume the agent and expert are the same or explicitly learn an action mapping

--> I am not sure what this could mean.

MoMaRT forms a more challenging task using a mobile robot

---> MoMaRT tackles a more challenging mobile robot task, namely ...

demonstrations are length-variant

--> demonstrations vary in length

Developing a policy to solve the FSI problem and simultaneously tackle these challenges is indispensable

--> indispensable to who? Perhaps you mean "Developing a policy to solve the FSI problem and simultaneously tackle these challenges is difficult but necessary for human-robot collaboration"

After receiving the rollout history h, the feature encoder extracts the history features f_h. Meanwhile, the feature encoder also extracts the demonstration features. Then, the task-embedding network computes the task-embedding f_ζ.

--> Is f_h the same as f_ζ. What is f even? A function is usually defined by a domain and a codomain before being used.

an adaptable error detector ϕ

--> again, no definition for ϕ

ϕ predicts the categorical probability y of the behavior mode for the lastest state in h

--> I am not sure what the behavior mode refers to. Also, "lastest state" --> "last state"

The task-embedding network is expected to handle length-variant sequences by padding frames

--> The task-embedding network is expected to handle input sequences of varying length with padding to a large prefixed length.

Our main experiment examines whether the AED methods can accurately report behavior errors when seeing the policy rollouts.

--> Our main experiment examines whether the method aimed at solving the AED problem can accurately report behavior errors when provided policy rollouts.

I am happy to help in identifying other statements that could use improvement but believe the above few represent a good starting point.

My concerns beyond the writing have been addressed by the authors response and my (fifth!) rereading of the paper. Perhaps my initial rating was a bit harsh, and hence, I've amended my rating to a 5 (although I am leaning towards 3). The problem is novel. The FSI tasks to evaluate Probe are comprehensive. But, since I believe that the writing needs a big update to be suitable for ICLR (and that has not been done so far by the authors), I am not recommending an accept considering the significant effort required by anyone to parse the first half (before experiments) of the paper.

Dear Reviewer Zod3,

Thank you for the valuable feedback and your time in reviewing our paper.

First, we sincerely appreciate your updated score and positive feedback on our work (related to the problem setting, method, and evaluation).

Besides, we agree that improving paper writing can strengthen our work and make it easier for all readers to follow. Thus, we are thankful for your writing suggestions in our earlier conversations.

To reflect our determination to continuously improve our paper writing, we have considered most of your suggestions and updated our paper. Please check the updated PDF (modified parts are marked as blue). We keep some parts as they are since we need more time to rephrase the paragraphs. But we will carefully inspect the whole paper and continually improve it.

Hence, if the updated PDF eases some doubts about paper writing and you recognize our problem, method, and evaluation, Please consider further updating your evaluation.

We will deeply appreciate it and try our best to address this remaining concern.

Thank you again for your valuable time to review our work and your active participation in the discussion.