Interactive Anomaly Detection for Articulated Objects via Motion Anticipation

Thank you for your positive feedback. We address your questions and concerns below.

1. Ablation for confidence-based loss and noise filtering

Table 2 (main paper) quantitatively and Figure A2 (appendix) qualitatively show the impact of multi-step noise filtering on the motion estimation performance. As further suggested, we also conduct ablation studies on the test categories to assess the individual contributions of the confidence-based loss and noise filtering strategy to our IAD task, using the AUROC metric.

2. Ablation for loss weights

The weights $\lambda_M$ and $\lambda_e$ in Eq. (6) are relatively insensitive and are therefore fixed to 1 across all experiments. We tune $\lambda_c$, the loss weight for the confidence loss, over the range [0.005, 1]. We find that a high value of $\lambda_c$ (e.g., close to 1) can cause training instability and slow convergence by dominating the primary motion loss, whereas a very low value leads to uniformly low confidence predictions across inputs. We set it to 0.05 for optimal performance. We will add discussion in the final draft.

3. It is unclear how much performance gain comes from better interaction versus better anomaly scoring

Our method consists of three integral steps: (1) learning an interaction (action), (2) motion modeling, and (3) anomaly scoring. Each step plays a critical role in the overall performance.

A good interaction agent can be helpful as it systematically explores the full articulation range of a part. Since functional anomalies are typically not visible without motion, the full articulation range of all parts needs to be thoroughly explored and tested to detect such anomalies.

While Where2Act [R1] offers a simple yet effective affordance-based interaction framework, it falls short in our specific IAD task due to its lack of motion-aware long-term planning. This limitation restricts its ability to fully explore articulation ranges, which is crucial for anomaly detection.

Learning an accurate motion estimator is equally important for our task, as it directly drives the final anomaly prediction by comparing against observed motion. Moreover, our motion estimator also facilitates long-term planning, enabling the selection of more informative future actions that lead to better exploration of the part articulations. In summary, our joint modeling of both action and motion is mutually reinforcing—better interaction policies lead to more informative motions, while accurate motion modeling supports more effective action planning.

The final step, our anomaly scoring as defined in Eq. (9), is straightforward and performs well provided the anticipated normal motion is accurate.

We also note that while more advanced interaction models may exist, Where2Act integrates seamlessly with our motion estimator, making it the most suitable baseline for our framework. Incorporating alternative models would require significant modifications to our motion model beyond the scope of this work.

4. Impact of noisy or partial depth observations

Our method inherently works well with partial point clouds as we train the model using depth observation from single viewpoint. Figure A1 (appendix) shows motion estimation results on real-world scanned articulated objects showing it can handle noisy input to a certain extent. To improve robustness, we can inject artificial noise to model real-world artifacts in the training depth data.

5. How critical is the use of the long-horizon motion estimation compared to atomic action steps?

Without long-horizon motion estimation, our model struggles with multi-step interactions. The lack of motion history often results in back-and-forth movements without progressing to new states. As shown in Table 1, our method outperforms baseline E, which uses atomic action steps with Where2Act [R1] as the interaction agent.

[R1] Where2Act: From Pixels to Actions for Articulated 3D Objects, ICCV'21

We appreciate your valuable feedback and recognition of our work's novelty. We address your points below.

1. Practical importance of the proposed IAD task

Thank you for your feedback. We agree that our proposed IAD task holds significant value for industries, which could benefit from inspecting digital twins of their manufactured products with movable parts. Those may include furniture (e.g., for cabinets) and consumer electronics (e.g., for laptops) industry for quality control and durability testing. It may also be used in product development and prototyping to identify design flaws and improve functionality, and, as well as, in service industry for inspecting and repairing articulated objects, and warranty assessments. We will add a discussion of this in the final draft.

2. Extendibility to real-world industry setting

We would like to clarify that our method does not require anomaly samples during training, which significantly reduces the effort of dataset construction and improves scalability.

Regarding extension to industrial settings, we note that many industries already maintain digital twins of their products as part of modern manufacturing pipelines. These digital twins can be used within simulation environments to generate rich interaction data. Recent advances in digital twin modeling and availability of advanced simulation tools (i.e., NVIDIA Omniverse) further support the generation of realistic, industry-standard datasets for IAD.

Furthermore, imperfections from real-world scans or digitization artifacts can be explicitly modeled during simulation, enhancing our model's robustness. Our method is general and designed to accommodate such variations. Future work can further incorporate additional physical properties such as friction to increase realism.

3. On threshold anomaly score

Following standard practice in AD benchmarks, our evaluation uses AUROC metric which is threshold-independent and obtained by varying over the different threshold values. A higher AUROC score means superior separability between normal and anomaly samples, i.e., stronger AD capability. In a practical setup, a fixed threshold is typically chosen based on the desired tolerance for false positives. This can be determined by calibrating on normal data to achieve an acceptable false positive rate (FPR).

Thank you for reviewing our work. Below we address your questions and weaknesses mentioned:

1. Concerns regarding novelty

Thank you for the feedback. We would like to clarify the novel contributions of our method compared to Where2Act [R1] as follows:

1. Where2Act focuses on predicting which actions are likely to move an articulated part, but it does not model how the part moves. In contrast, our method estimates the rigid motion flow that fully characterizes the part’s kinematic response to the action. This is crucial for our interactive anomaly detection (IAD) task, as it provides the anticipated ‘normal’ motion necessary for comparison with the observed motion.
2. Our formulation enables long-horizon, temporally consistent action exploration, which Where2Act lacks.
3. By explicitly estimating motion, our method can filter out noisy actions by identifying inconsistencies in predicted motions.
4. Our model leverages rich supervisory signals from simulation during training, including motion mask and rigid motion matrix tied to each input action, whereas Where2Act uses only discrete supervision (motion vs. no motion). Our richer supervision allows to capture the continuous range of articulation dynamics more accurately.

Comparison with autonomous driving trajectory anomaly detection: In autonomous driving, anomaly detection is typically based on passive observation of agents (vehicles or pedestrians) moving along structured paths governed by traffic rules. These methods rely on detecting spatial deviations in smooth trajectories. In contrast, our task involves an embodied agent actively manipulating articulated objects, where anomalies arise from deviations in expected part dynamics (e.g., stuck or broken joints). This demands modeling of contact-driven motion, causality, and part-specific behavior. We will discuss about the related papers in the final draft.

We would like to highlight that other reviewers have acknowledged the novelty of our work. For instance, Reviewer dCsY acknowledged our joint modeling of interaction and motion anticipation as a novel approach, Reviewer bQ9u remarked that our proposed method is well-posed and clearly articulated, and both reviewers recognized the novelty and practical importance of the IAD task itself.

[R1] Where2Act: From Pixels to Actions for Articulated 3D Objects, ICCV'21

2. Selection of training categories

Our chosen category split follows the same protocol as in [R2]. As suggested, we also conduct experiments with different combinations of training categories. While selecting these combinations, we ensure that each combination contains a sufficient number of articulated objects covering representative articulation types, such as both revolute and prismatic joints. The StorageFurniture category is included in all sets as it is the most representative category in the PartNet-Mobility dataset. The test categories remain consistent across all experiments. As shown in the results below, our choice of training categories has a relatively low impact on overall performance, although some test category might benefit from the presence of functionally similar category in the training set (e.g., Microwave and Safe as in SetA). Note, while the overall performance is slightly lower than in Table 1, this is expected since training was conducted on a smaller subset of the data.

[R2] UMPNet: Universal Manipulation Policy Network for Articulated Objects, ICRA'22

3. Anomaly generation details

Our dataset contains a broad range of plausible functional anomalies covering types such as: tilted axis of motion, wrong joint type, restricted motion etc. A detailed description of the anomaly generation process is provided in Appendix A.1. Currently, the dataset focuses on 1-DoF prismatic and revolute joints due to limitations in publicly available articulation datasets, such as PartNet-Mobility, which includes only these joint types. However, our methodology uses a general motion parameterization that is agnostic to joint type and can be extended to more complex articulations.

4. Additional metrics

Our evaluation set is relatively balanced (see Table A1 in appendix). As suggested, we also report AUPRC and FPR@95% recall on the test categories in the table below.

5. On clarity and inconsistent terminology

Thanks for your feedback. We will fix the inconsistencies in the final draft.

6. Analysis of failure cases

In addition to the analysis in Section 4.4, Figure A.3 (appendix) presents more examples of representative failure cases. The primary cause of failure is due to inaccurate motion prediction that manifests in several ways, such as misclassifying the joint type (e.g., predicting rotation instead of translation), defining an incorrect axis of motion, or generating physically implausible movements that ignore part collisions. Please see Section A.3 (appendix) for more discussion.

7. Explanation of Online data sampling

While offline data generation using random actions can partially explore the action space, it is often inefficient due to the large size of the action space. Focusing on actionable interactions (i.e., actions likely to induce motion) yields more informative training data. Online data sampling provides an efficient way of collecting actionable interaction data by leveraging the knowledge already acquired by the network. Specifically, we sample actions using the normalized movability score distribution in Eq. (8) and simulate additional interactions for those high-likelihood actions. We skip noise filtering in this step to sample actions faster, since this process is not sensitive to outliers. This online data collection strategy significantly increases the proportion of interaction pairs that induce positive motion.

Thank you again for your valuable comments and feedback. We hope that our detailed rebuttal has addressed your concerns. Please let us know If there are any remaining questions or points you would like us to clarify, we would be happy to address them during the remaining rebuttal period.

Dear Reviewer osxE,

Could you kindly clarify which of your comments you feel have not been adequately addressed? Additionally, could you specify the areas where the paper still requires significant revision?

I encourage both the reviewers and the authors to actively engage in the discussion to help move the process forward constructively.

Best regards,

AC

We thank Reviewer osxE for participating in the discussion.

For the benefit of the AC and the final discussion, we emphasize that our paper’s contributions extend beyond the proposed dataset, with our primary contributions as follows:

1. A Novel Task: The formulation of Interactive Anomaly Detection (IAD) for articulated objects, a new problem space that requires active agent interaction.

2. A Novel Method: A new, learning-based approach that uniquely integrates interaction and motion anticipation to solve this task. We have detailed its novelty and significant differences from prior work like Where2Act in our rebuttal.

3. A New Benchmark: The creation of the PartNet-IAD benchmark, which is essential for evaluating methods on this new task, but is not the sole contribution of our paper.