Every Mistake Counts: Spatial and Temporal Beliefs for Mistake Detection in Assembly Tasks

We appreciate the reviewer's recognition of our paper's significance in addressing ordering mistakes in assembly tasks and kind comments on our data, new labels and proposed framework.

---

W1. The paper does not discuss any limitations.

A1: Thanks for this feedback! We acknowledge that, in its current form, our current approach doesn't generalize well to new object parts and is heavily influenced by perception quality. We'll incorporate these points into our revision.

W2. The paper lacks a better exposition (text quality, mathematical notation, figures), which makes it difficult to fully assess the contribution.

A2: We appreciate the reviewer's feedback and will clarify and refine our contributions in our paper and will enhance its clarity.

W3. Missing ablation studies.

A3: Unfortunately, our framework does not allow for various ablations. In Table 2, we've explored various perception module levels to show the gap between GT and computer vision-based perception modules.

---

Q1. Section 3.2: What is meant by 'given by definition as $A_{ij}$?

A1: $A_{ij}$ is defined in Sec. 3.1 and indicate that the action of attaching $i$ and $j$ is a correct action.

Q2. The in-text description of Figure 2 refers to 'edges', but no edges are shown in Figure 2 (left). Does this mean the graph has no edges?

A2: Thanks for bringing this to our attention! We’ll revise the corresponding text to refer to Figure 2 (right), when discussing edges,

Q3. Section 3.2: The text states that 'completion occurs when the episodic context M fully traverses the graph'. Does this refer to the rule graph?

A3: Yes, your understanding is correct! It does refer to the rule graph.

Q4. What is a 'focal component pair'?

A4: The term is to denote the part pairs of interest that the temporal rule is designed for as described in Sec3.3 (the second paragraph).

Q5. The mathematical notation appears convoluted and more difficult to follow that it seems necessary. After reading Section 3.2 and 3.3, it was not clear how Spatial and Temporal beliefs are defined.

A5: Sec 3.2 explains that spatial belief is defined as the allowed connection between parts, which is also considered as the edge of the rule graph. Sec 3.3 essentially talks about two types of temporal beliefs and how they can be used to infer ordering mistakes.

Q6. Given the quality of the exposition, I have the impression that this paper was rushed and that the text needs to be refined prior to publishing this work.

A6: We thank the reviewer for the comment, and we are actively working on revising the text to address the concern and enhance the overall quality.

Q7. Section 4.3: What are 'fine-grained' mistakes?

A7: The coarse mistakes only care about mistakes, correct and corrections without accounting for their root causes. Following this, the fine-grained mistakes are introduced in Sec 4.2 (the last paragraph on page 7), which include generic ordering mistakes (B1), accumulated mistakes (B2), unnecessary detachment (B3), and Misorientation mistakes (B4). In addition, we provided more information on the mistake types in our supplementary.

Q8. To assess the performance of the Inferences vs the BeliefBuilder, I would appreciate ablation studies.

A8: The BeliefBuild essentially builds the rule graph which is later used by the Inferencer during testing only. We therefore cannot compare the Inferencer vs the Beliefbuilder.

We appreciate the reviewer's thoughtful feedback, and thank for recognizing our paper's topic and how challenging it is and our design for temporal belief modeling.

---

W1. It is not clear whether the input is a sequence of images or labels representing the actions. Moreover, if the part-to-part annotations and the synthetic dataset are treated as contributions, then more information should be included in the main paper, rather than the supplementary material. Please add a couple examples of the synthetic sequences with mistakes.

A1:

• Yes, our approach generally takes as input labels and such labels can be obtained from multiple sources, for example, the video annotation, parsed from narration in the video, or the output from a perception module. In this work, we evaluate our approach with both annotation and perception outputs in Table 2.
• Thanks for highlighting the synthetic data issues. We'll add the examples to the main paper for revision.

W2. In my understanding, the BeliefBuilder is creating an explicit rule graph that only applies to a specific object and its set of components. Aren't explicit knowledge grounded beliefs very limited in terms of generalization? Wouldn't it be more useful to learn a soft graph with latent representations that could potentially be applied to new objects?

A2: We haven't explicitly tested for generalization, but our model is expected to perform reasonably for shared part pairs like interior-cabin and sound module-roof on different toys. However, for novel object parts across toys, prior knowledge is necessary, e.g., informing the model about the similarity of attaching a dumper vs a container which are usually positioned in the rear part of the toy. One option to get such information could be through leveraging LLMs, so that we can generalize to new objects like unknown containers by recognizing structural similarities aligned with linguistic semantics. We thank the reviewer and will incorporate these discussions into our paper.

Q1. What is the performance of the Inferencer on Seen vs Unseen objects?

A3: As we mentioned above, we focus on seen objects in this paper, but with structural prior knowledge, our rule graphs can be extended to unseen objects.

We appreciate the reviewer's positive feedback on the proposed spatial and temporal beliefs and our improved performance over baseline methods.

---

W1. The paper is hard to read and follow. The system is described in a complicated way, though the underlying mechanism is straightforward.

A1: Thank you for your comments. Even though the underlying mechanism appears straightforward at first glance, unraveling the less apparent challenges within ordering mistake detection requires a detailed and comprehensive description. We will improve the readability and ease of comprehension to ensure our paper is accessible to a broader audience.

W2. The paper only tackles ordering assembly mistakes, which is just one type of failures. LEGO construction tasks are also excluded.

A2: We agree that solving LEGO and other assembly mistakes are interesting. For a scientific paper, however, the primary aim often revolves around delving deep into a specific problem setup rather than addressing the broader spectrum of challenges involved in product development. While the paper might focus narrowly, its insights can contribute to a broader understanding of the field. LEGO and other assembly mistakes are out of scope, as we explain below.

• The Assembly101 dataset breaks down the mistakes - ordering mistakes comprise 2/3 of the overall mistakes. The remaining third are 3D placement mistakes. Solving these mistakes requires 6D pose estimation of the object parts and is a vastly different problem than our focus on temporal reasoning and outside our scope.
• We exclude LEGO because the underlying problem setting is fundamentally different and we explain this explicitly in our manuscript. LEGO bricks are designed to allow creative freedom to allow open-ended construction. Without brick semantics, mistakes mainly arise from imprecise attachments, which require tactile feedback that is an entirely different field much less beyond the scope of this work.

W3. The method avoids perception and reasoning but instead resorts to human annotations for the parts and the steps, which is an unrealistic simplification of the problem.

A3: We respectfully disagree. Our work simply separates perception and reasoning into two separate components, and two are developed independently.

• Reasoning: the separation of reasoning from the perception is standard in robotics research [A]; Mistake detection is essential for many applications, and waiting for perfect perception to be solved is not a practical approach for advancing research.

• Perception: This is not our main focus so we rely on existing model (TSM), which we train for our problem setting. We verify that using perception-based outptus integrates seamlessly in our approach as shown Table 2. While there are some performance drops due to imperfect perception, we don’t think this takes away from the underlying contributions of our proposed work.

[A] Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement. In ICRA, 2023

Questions

Is it really difficult to integrate state of the art perception with this pipeline? If so, why?

A: In this paper, we decouple the perception and reasoning modules. We can utilize the output of the perception module, simply by replacing the GT labels with the output of the perception module which are expected to be the per-frame/segment action labels, as seen in Table 2.

It would be really helpful if you could provide the end-to-end results of this method and state-of-the-art (SOTA) with perception.

A: We integrate our mistake detection module with the perception module's outputs in Table 2. Notably, when AR Acc is 86.4%, errors in the perception module result in a decrease in mistake recall to 42.3%, highlighting the module's impact.

A real-time video demonstration showing the ordering mistake detection in the assembly task, using only the RGB frame, would be very intriguing.

A: We will provide these for both ground truth labels and perception module outputs.

Is it not possible to create and evaluate this method and SOTA on a larger dataset with different objects, more complex sequences and more parts? If so, what are the difficulties? A larger dataset would alleviate the concerns of generalizability.

A: The primary challenge in adapting our approach is the scarcity of datasets with mistake labels. Assembly101 is the only existing large dataset providing such labels. To address this, we created the synthetic EGT dataset, which our model performs well on. However, the lack of alternative mistake datasets limits further testing and validation on real data.
We can in principle adapt our approach to more complex sequences and more parts, which may require to account for the hybrid type of temporal belief as introduced in our supplementary.

If perception is included in this pipeline and you are able to estimate the object pose, how difficult would it be to extend this method to detect orientation errors?

A: Orientation is independent of temporal reasoning and can be addressed as a frame-wise issue through object pose prediction. We expect that with an accurate object pose prediction method, detecting orientation mistakes is feasible. To avoid confusion, we are considering renaming our paper to clarify that we focus on detecting ordering mistakes.

We appreciate the reviewer's kind words on our well-timed framework for a previously overlooked challenge and our clear and organized presentation.

---

Trivial task definition

A: We focus on ordering mistakes in this paper, which comprise 2/3 of overall mistakes in Assembly101 and require temporal reasoning. We do agree with the reviewer that object pose prediction and action recognition are worth investigating as challenging tasks in the computer vision domain. We refrain from shifting our attention to such aspects in this work for the following considerations:

• Object pose prediction [A] primarily focuses on large objects, requiring object meshes for seen classes for some models; Assembly101 lacks these meshes. Additionally, studying orientation issues is a frame-based investigation problem, which is also orthogonal to our temporal reasoning for the ordering mistake detection.
• Action recognition on the well-studied egocentric dataset Epic-Kitchens reaches only 56% accuracy [B]. Research [C] on adverbs like "tight" modifying the verb "fasten" has been limited to small-scale studies, achieving 71.9% accuracy for just 5 adverbs.

Perception is ignored

A: Perception is an orthogonal line of work that requires more progress until we can fully realize a vision-based intelligent assistant which can simply observe our setting and respond.

• On one hand, we utilize GT labels to isolate mistake detection from perception module errors, which provides a unbiased presentation of the ordering mistake detection problem.
• On the other hand, we did not ignore perception as we integrated the existing action recognition outputs with our model and reported the results in Table 2.

Table 2 shows that our accuracy decreases due to accumulated perception errors, highlighting the need for improved perception to enhance mistake detection. By using GT labels, we can continue to advance research on mistake detection without being hindered by limitations in CV-based recognition, enabling progress despite these challenges. This is similar to how QR codes and RFID tags aid robot navigation, providing identifiable markers to tackle the limitations of the perception module.

Limited evaluation

A: We politely disagree that our approach is tailored for the toy assembly.

• Our spatial and temporal beliefs are generic, not specific to toy parts; as we tested them on 101 different toys. Through synthetic experiments in our paper, we demonstrate the adaptability of our approach for EGT assembly in industry settings, serving as a proof of concept for its potential to generalize and improve assembly tasks across various domains.
• Assembly101 is the sole dataset with these annotations. And it encompasses intricate assembly scenarios, averaging 14 steps, making it a valuable resource for studying mistake detection and learning. Even adults assembling children's toys make numerous mistakes, highlighting the complexity of the task and the need for improved learning strategies.

Limitations in the approach

A: We acknowledge that our approach is based on logic which build the spatial and temproal knowledge, however, we respectfully disagree with the reviewer that the inclusion of our work in ICLR is of less importance. Specifically, our spatial and temproal beliefs can be of great interest to the audience working with robotics, planning.

Limitations in the logic

A: The reason that our approach is not 100% accurate is due to the training data. Our approach's accuracy is limited by training data constraints. When we only have access to a subset of error-free action sequences, it will hinder our algorithm's ability to handle mistakes during testing. We performed our algorithm over 4 splits to account for this.

Minor: Clumsy Notations, Clarity in Fig. 3 and Fig. 5.

A: Thanks for the comment.

• We do state everything in words, either above or below the equations. We found that the equations provide a precise and concise way to present complex relationships, making the information more unambiguous. They enable precise communication of ideas which can be challenging to convey in words alone.
• Thanks for pointing this out and we’ll highlight the differences and update in the figures.

[A] BOP Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objects, CVPRW 2023

[B] Training a Large Video Model on a Single Machine in a Day. arXiv preprint arXiv:2309.16669 (2023).

[C] Learning Action Changes by Measuring Verb-Adverb Textual Relationships. in CVPR. 2023.