AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation

[Part 1/2]

We thank the reviewer for their feedback. We appreciate your recognition of AHA as an approach that can be scaled up to other scenarios, and for acknowledging the crucial role that failure reasoning plays in downstream policy learning, which could be essential for advancing the field. All changes in the revised paper and supplementary materials have been highlighted in blue for easy reference.

Below, we address your comments, feedback, and clarification questions:

1. Clarity in implementation

The implementation details of the baselines are unclear. What type of input data are the baselines using? Is it a single image, a sequence of images, images from multiple views, or video? This distinction significantly affects baseline performance. For instance, Gemini excels at reasoning for robotic manipulation when using video inputs, but its performance would likely degrade if multiple-view images were used instead.

For all the proprietary and open-source models evaluated, we ensure consistency by using the same input images and prompt structure, as illustrated in Table 2 of the Supplementary Material. The layout of concatenated frame images varies across datasets due to differences in the number of views and other dataset-specific factors. For AHA, the input includes a sequence of images from three distinct views, with the sequence length varying depending on the task. Additionally, Maniskill-Fail uses three views, while RoboFail relies on a single view. While we recognize that different models may perform better with specific types of inputs, our primary objective is to enable a fair comparison by providing all models with the same contextual information.

[Part 2/2]

2. Additional baseline and ablation studies

Lack of Comparison with Robotic-Specific Methods: Why not compare your approach with other robotic-oriented reasoning methods mentioned in the 'Introduction' or 'Related Work' sections? The current experiments only compare against general Vision-Language Models (VLMs), which are not specifically designed for robotic tasks.

We appreciate the suggestion to compare our work with the studies mentioned in the related section. Many of these works utilize off-the-shelf Vision-Language Models (VLMs) or Large Language Models (LLMs), prompting them with task scene state information to perform failure detection and reasoning. This approach differs fundamentally from AHA, which is the first VLM specifically instruction-tuned for failure detection and reasoning. To address this suggestion, we implemented REFLECT, a failure reasoning system that also leverages off-the-shelf LLMs for reasoning. However, REFLECT incorporates state information obtained from a hierarchical scene graph constructed using multimodal sensory inputs. Despite these fundamental differences, we evaluated AHA and REFLECT on the RoboFail subset and found that AHA outperformed REFLECT in three out of four evaluation metrics, including binary success, LLM fuzzy matching, and cosine similarity.

View Angle Concerns: The camera view angles during training are fixed according to the RLBench settings. This raises concerns about the view angles during testing. Specifically: 1) How many views does the model require during testing? 2) Should the view angles during testing match the exact poses used in the RLBench training setting?

We thank the reviewer for raising this point. While AHA is trained on RLBench settings, our results in the main paper Table.2 demonstrate that the model can effectively reason in test scenarios where the viewing angle differs from those in RLBench and further supported with more ablation studies in supplementary material section 1.5. Additionally, we evaluated the AHA-trained model using different numbers of viewpoints during inference. The performance, measured by ROUGE, Fuzzy Matching, and Cosine Similarity scores, remained consistent whether 1, 2, or 3 views were used. This robust generalization likely stems from the reasoning capabilities inherent in the co-training data.

Missing Reference to Figure 1: Figure 1 is not referenced or discussed anywhere in the paper, which needs to be addressed for clarity and completeness.

Thanks to the reviewer for pointing this out. We have added a reference to Figure 1 on line 103 of the paper.

We sincerely hope that by addressing the reviewer's concerns and providing the necessary clarifications, our paper may be considered for an improved rating. If there are any further questions, please do not hesitate to let us know. Thank you for your time and thoughtful review.

Dear Reviewer qHQf,

Thank you for your thoughtful feedback and for recognizing the significance of failure detection and reasoning. As the rebuttal phase concludes, we kindly invite you to review our response. We have addressed your comments by:

• Improving clarity in writing,
• Adding experiments with REFLECT as a new robotics-specific baseline,
• Conducting ablation studies on how the number of viewpoints impacts AHA's performance.

We hope these updates address your concerns. If so, we would greatly appreciate it if you could consider raising our scores.

Thank you again for your valuable feedback and support.

We thank the reviewer for their feedback. We appreciate the recognition of AHA as an interesting idea and see novelty in our approach of using AHA’s failure detection and reasoning to improve VLM-based robotics applications. Furthermore, we are grateful that the reviewer notes the importance of our AHA datasets for building robust large models that can better understand the physical world. All changes in the revised paper and supplementary materials have been highlighted in blue for easy reference.

Below, we address your comments, feedback, and clarification questions:

1. Question regarding details in the paper

What is the difference between rotation and no rotation? What are the details of your baseline settings? I couldn't find them in your paper.

We acknowledge that our phrasing may have caused some confusion. The two failure modes differ as follows:

Incorrect Rotation Failure This is an active failure mode where the robot reaches a keypoint with a rotation offset, which eventually leads to task failure.

Missing Rotation Failure This is a more passive failure mode, where the robot is expected to adopt a new rotation at a keypoint but instead retains the rotation from the previous keypoints, resulting in failure.

To address this, we have rephrased the wording in Section 3.1 for improved clarity. Additionally, as we evaluate our VLM’s capabilities alongside both open-source and proprietary VLMs, we have aligned the input prompts and system configurations for consistency. For more details on this alignment, please refer to Table 2 in the supplementary material.

2. Additional Baseline

REFLECT seems highly relevant to the work, and would like to see it as a baseline.

We thank the reviewer for suggesting the implementation of REFLECT as a baseline. In response, we implemented REFLECT on a subset of the RoboFail evaluation dataset and compared it with our method, AHA. Due to REFLECT’s reliance on multimodal sensory inputs such as audio—which are not available for other evaluation datasets—we limited the comparison to this subset. It is important to note the inherent differences between the two approaches. Our method involves instruction-tuning a Vision-Language Model (VLM) for failure detection and reasoning, while REFLECT uses a visual prompting technique that leverages off-the-shelf VLMs. REFLECT relies on detailed task and state information extracted from a hierarchical scene graph, which is constructed using diverse multimodal sensory inputs. Despite these differences, we evaluated AHA and REFLECT on the RoboFail subset and observed that AHA outperformed REFLECT in three out of four evaluation metrics, including binary success, LLM fuzzy matching, and cosine similarity.

3. Future plans for scaling AHA

How can this work be scaled up and extended beyond its existing failure type.

We acknowledge the current limitations of our approach to failure generation, which is conducted in a systematic manner focused on a predefined set of failure modes observed from existing robot datasets and policy rollouts. This method, while effective, does not encompass all potential failure modes and edge cases. As part of our future work, we plan to expand this approach through various methods, such as systematically distilling failures from policies using a real-to-sim setup or extracting latent representations of failures from real-world human videos. This enhancement seeks to improve the generalizability of our method beyond its current scope, enabling it to capture a broader range of failure scenarios.

We sincerely hope that by addressing the reviewer's concerns and providing the necessary clarifications, our paper may be considered for an improved rating. If there are any further questions, please do not hesitate to let us know. Thank you for your time and thoughtful review.

Dear Reviewer FdiA,

Thank you sincerely for your time and thoughtful feedback on our paper. As the rebuttal phase nears its conclusion, we kindly invite you to review our submitted response. We have worked diligently to address your comments, including:

• Clarifying terminology used in the paper,
• Adding a new baseline with REFLECT and comparing results with AHA,
• Expanding on future directions and potential extensions of AHA.

We hope these revisions address your concerns thoroughly. If you find the updates satisfactory, we would be grateful if you could consider raising our scores.

Thank you again for your valuable feedback and support.

We thank the reviewer for their feedback. We appreciate the recognition of AHA as an interesting idea and see novelty in our approach of using AHA’s failure detection and reasoning to improve VLM-based robotics applications. Furthermore, we are grateful that the reviewer notes the importance of our AHA datasets for building robust large models that can better understand the physical world. All changes in the revised paper and supplementary materials have been highlighted in blue for easy reference.

Below, we address your comments, feedback, and clarification questions:

1. Soundness in approach

How can we guarantee that AHA improves some of these downstream applications, such as VLM-based reward design? Is it possible to observe iterative reward tuning behaviours as a result of failure feedback?

We appreciate the reviewer’s question. While we do not claim that AHA will definitively improve downstream applications such as VLM-based reward design, we propose that failure reasoning modules like AHA could potentially benefit such applications. This is supported by the empirical experiments presented in the paper. It is important to note that our primary goal is not to provide a complete solution for reward design using failure feedback and reasoning but rather to demonstrate that this is a viable use case for AHA. To further illustrate this, we have included examples of AHA's application in iterative reward tuning for VLM-based reward design on the anonymous project page (https://aha-iclr.github.io/) for reference. Based on our observations, when AHA consistently succeeds in accurately reasoning about failures for the given tasks, it effectively guides the RL policy toward the optimal dense reward function.

2. Clarity in writing

It would be more convincing to show more extensive comparisons of GPT4o against AHA. This would help in understanding if GPT4o performs poorer than AHA due to the context length of its output or other potential factors. One way to analyze that is to have a human oracle upper bound.

Generalization in approach

We apologize for the lack of clarity regarding the downstream tasks in our comparison between AHA and GPT-4o. Our experiments indicate that GPT-4o's poor performance was not due to the length of the failure descriptions but rather its inaccurate failure reasoning, which negatively impacted downstream task performance. To address this, we included example GPT-4o outputs for failure reasoning in the supplementary material Table 4. Additionally, based on the reviewer's feedback, we incorporated human-oracle evaluations with diverse writing styles. We recruited participants to curate and evaluate failure detection and reasoning outputs on the RoboFail dataset. These outputs were assessed using the four evaluation metrics, providing an upper performance bound for comparison with AHA. Although there is an obvious gap between AHA and human-oracle, AHA is still the SOTA in failure reasoning in comparison with other approaches.

How to make AHA more generalizable for other forms of robotics task reasoning?

While we acknowledge that most downstream tasks in this paper focus on tabletop manipulation due to the nature of the fine-tuning data used for VLM training, we have tested AHA across various embodiments, viewpoints, tasks, and domains—even using randomly sampled manipulation videos from the internet for failure reasoning. Generating failure trajectories from simulation, one of our main contributions, and this is broadly applicable across domains. This data generation framework can be extended to domains such as locomotion, navigation, and dexterous manipulation, which we plan to explore in future work. The key challenge lies not in the methodology for failure generation and VLM fine-tuning presented in this work, but in adapting the failure dataset generation process to support broader domains. We are looking to expand this approach to systematically distill failures from policies using a real-to-sim setup, aiming to enhance the generalizability of our method beyond its current scope and will leave it for future work.

We sincerely hope that by addressing the reviewer's concerns and providing the necessary clarifications, our paper may be considered for an improved rating. If there are any further questions, please do not hesitate to let us know. Thank you for your time and thoughtful review.

Dear Reviewer Enxm,

Thank you for your time, thoughtful feedback, and for recognizing the novelty and interest in our work. As the rebuttal phase concludes, we kindly invite you to review our response. We have worked diligently to address your comments, including:

• Enhancing clarity in writing,
• Providing further insights on how AHA can be extended,
• Demonstrating AHA's usefulness in iterative reward guidance scenarios,
• Incorporating a human oracle to evaluate VLM sensitivity to context length.

We hope these revisions thoroughly address your concerns. If you find the updates satisfactory, we would greatly appreciate it if you could consider raising your scores for our paper.

Thank you again for your valuable feedback and support.

We thank the reviewer for their feedback. To address the concerns thoroughly, we conducted an additional human oracle study on the downstream task (VLM reward design). In this study, human participants watched video rollouts after each iteration of training, provided natural language reasoning on the failure feedback, and grounded this reasoning to improve dense reward function generation. The results are as follows:

Based on these results, we observe performance comparable to AHA when a human user provides natural language reasoning to improve VLM reward design. However, this performance can vary depending on the complexity of the RL tasks. Additionally, our evaluation was conducted with a fixed number of iterations rather than until convergence, which can differ for each task. However, based on the result, we do see that better failure reasoning helped with improving downstream tasks in general.

Dear Reviewer Enxm,

Thank you for your thorough review and valuable feedback, which have greatly helped us improve the paper. We also sincerely appreciate that you recognize the interest, novelty, and importance of our work. Our work presents several novel contributions to the field:

1. The procedural failure generation pipeline for producing labeled failure demonstrations for instruction-tuning Vision-Language Models (VLMs).

2. The first instruction-tuning of a VLM to detect and reason about failures, demonstrating superior performance compared to open-source and proprietary VLMs across tasks, embodiments, and domains.

3. A range of use cases illustrating the versatility and potential applications of our approach.

We kindly ask you to consider these contributions in your evaluation. Additionally, we have conducted an additional experiment using a human oracle for the VLM-based reward design tasks which you have requested, we hope you could have a reconsideration of our work.

Please let us know if there are any further questions or concerns we can address.

Thank you for your time and thoughtful consideration.

We would like to thank the reviewer for the follow-up. We wanted to clarify a few points:

Even when the VLM is provided with (ground-truth) human-generated failure descriptions, it still struggles to design rewards that achieve even an 80% success rate.

We would like to clarify that our VLM (AHA) does not directly design rewards for methods like EUREKA. Instead, AHA detects failures, reasons about them, and passes the insights to a proprietary LLM (GPT-4o) for reward design, as mentioned in L481-482 of the paper. Therefore, the results demonstrate that AHA enhances the success rate of downstream tasks leveraging EUREKA by providing more accurate failure reasoning, compared to the default EUREKA reflection mechanism (replaced with the proprietary VLM) as shown in Figure 3 (right) in the paper.

This indicates that relying on failure descriptions to enhance a VLM's reward generation capability is inherently limiting.

We acknowledge that achieving a 100% success rate remains a challenge. However, we emphasize the significant 22.31% improvement over the baseline, demonstrating a substantial performance gain.

Moreover, AHA appears to have already surpassed human performance in reward design, suggesting that the proposed approach cannot be further improved by incorporating seemingly higher-quality labels.

We recognize that reward design incorporating AHA's failure reasoning occasionally surpasses reward design using human feedback. However, we argue that humans should not be considered the upper bound for reward design. Effective reward shaping often involves more nuanced insights, as evidenced by the EUREKA paper. For example, in Figure 8 of their work, the authors show that purely human-based reward shaping performs poorly compared to LLM-aided approaches. This aligns with our findings, even across very different tasks in Eureka paper, whereby leveraging a VLM like AHA could potentially achieve comparable or occasionally superior performance to manual human feedback.

especially considering that the tasks in experiments are already solvable through imitation learning and hand-engineered RL rewards.

We respectfully disagree with the reviewer on this, both imitation learning and hand-engineering for RL rewards require lots of efforts from human experts in robotics to collect demonstrations or design dense rewards. AHA, combined with Eureka, remove the requirements for human expertise/human intelligence to solve robotic tasks hence making both work a contribution to the field.

its utility for various downstream applications remains uncertain

We appreciate the reviewer acknowledging our contribution to failure reasoning as a dataset. However, we respectfully disagree with the claim that our contribution lacks utility for downstream tasks solely because these tasks are already addressed by existing methods. Firstly, our work extends beyond helping with improving VLM-based reward design, we also shown that AHA can help with improving LLM-based task planning for TAMP systems or enhanced task verification for zero-shot manipulation systems. With the increasing adoption of off-the-shelf LLMs and VLMs in robotics, AHA's failure reasoning and feedback have the potential to support a wide range of future tasks and downstream applications.

Given these clarifications, we kindly hope you will take them into consideration when re-assessing our work.

[Part 1/2]

We thank the reviewer for their thoughtful and detailed feedback, which reflects their significant effort. We appreciate the reviewer's recognition of the importance of our work for safety-critical applications when deploying learning-based robotics algorithms in the real world, and acknowledge the importance of failure detection and reasoning. We are pleased that the reviewer found our paper well-structured, easy to follow, and that our results are comprehensive and informative. We have made substantial efforts to revise the paper's presentation and methodology, and have included additional experiments as suggested by the reviewers, all aimed at improving the paper's overall quality. All changes in the revised paper and supplementary materials have been highlighted in blue for easy reference.

Below, we address your comments, feedback, and clarification questions:

1. Clarity in writing

It would be clearer if subsection 4.1. can be shortened a bit and described with a more concise mathematical formulation. Furthermore, adding a subsection in method to explain how improved failure detection and reasoning performance can help these downstream tasks.

We thank the reviewer for the advice, and have revised Section 4.1 to include a more concise mathematical formulation. Furthermore, we have added Section 4.4 as a new subsection to provide more explanation on how AHA can help with these downstream tasks.

Fig.1 and Fig.2 are hard to read. It would be better to reorganise the sub-figures and increase the sizes of the images and texts.

We thank the reviewer for the feedback on the figures and have made the necessary adjustments to Figures 1 and 2 to enhance their readability.

2. Questions on Methodology

How does the proposed fine-tuned VLM address hallucination issues in failure detection, and what is the impact of fine-tuning with the dataset on uncertainty estimation and the trade-off with prediction accuracy? Additionally, there is missing citation.

We thank the reviewer for the suggested citation. We have added the missing references and discussion in Lines 143–145, acknowledging that our method does not fully address hallucination in LLMs/VLMs. However, fine-tuning on domain-relevant data has shown improved performance in robotics scenarios. To highlight the importance of fine-tuning with our proposed dataset, we examined the correlation between AHA's prediction accuracy and its uncertainty estimations. Specifically, we compared sentence token prediction probabilities from AHA-13B with those from LLaVA v1.5-13B. AHA-13B demonstrated significantly higher average prediction probabilities, aligning with its superior accuracy. These findings, which indicate a positive impact of fine-tuning with the AHA failure dataset, are included in the supplementary material section 1.5.

How does CoT prompting or other prompting templates help with the baselines so as to compare with AHA.

We appreciate the reviewer’s suggestion that Chain-of-Thought (CoT) prompting or other advanced prompting methods could potentially improve baseline performance, such as GPT-4o, beyond the in-context learning results already presented in the original paper. To investigate this, we implemented Multimodal Chain-of-Thought Reasoning (MMCoT) [1] and evaluated it with GPT-4o on the RoboFail dataset. Despite these enhancements, our approach continues to outperform MMCoT on GPT-4o, demonstrating its effectiveness in failure detection and reasoning.

[1] Zhang, Zhuosheng, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. "Multimodal chain-of-thought reasoning in language models." arXiv preprint arXiv:2302.00923 (2023).

[Part 2/2]

3. Additional Ablation Study

It would be insightful to understand how varying the number of camera viewpoints affects downstream tasks.

We thank the reviewer for suggesting an ablation study on the impact of camera viewpoints for failure reasoning in downstream applications. To empirically evaluate how viewpoints affect AHA's performance, we conducted zero-shot evaluations on the ManiSkill-Fail dataset. Specifically, we compared three input configurations: single-viewpoint images, concatenated two-viewpoint images, and the original dataset configuration with three concatenated viewpoints. Our results indicate a slight performance advantage when using single-viewpoint images. We attribute this to the resolution limitations of the LLaVA-1.5 visual encoder (256x256), where single-viewpoint inputs provide clearer visual information for failure reasoning. The ablation results are attached for reference:

Due to missing real-world experiments, therefore, it is hard to evaluate the performance loss during sim-to-real transfer and the actual practicality of the proposed technique.

We acknowledge the limitation of not including real-world experiments in our work. This decision stems from the significant differences in setup across the three downstream tasks (Eureka, Trust the Process, and Manipulate-Anything). However, we provided quantitative results in simulation, conducted in the same manner as in the original papers for these tasks. Each of those papers includes its own demonstration of real-world experiments and the feasibility of sim-to-real transfer, supporting the applicability of our approach in real-world settings. The primary focus of AHA is to demonstrate how a Vision-Language Model (VLM) can be fine-tuned for failure detection and reasoning, showcasing its effectiveness and its potential for a wide range of downstream robotics applications. These applications are not limited to the three examples presented in the paper but extend to broader use cases in the field. Lastly, sim2real for these downstream tasks are not part of our contribution of the paper.

We sincerely hope that by addressing the reviewer's concerns and providing the necessary clarifications, our paper may be considered for an improved rating. If there are any further questions, please do not hesitate to let us know. Thank you for your time and thoughtful review.

Dear Reviewer zG7S,

Firstly, we sincerely appreciate the time and effort you have dedicated to reviewing our paper. As the rebuttal discussion session draws to a close, we kindly request you to review our submitted rebuttal. We have made significant efforts to address your comments, including:

• Reorganizing Section 4.1 and adding a subsection in the Methods section to clarify how improved failure detection and reasoning contribute to downstream tasks.
• Incorporating the missing citation you highlighted and conducting additional experiments to analyze AHA's prediction likelihood versus accuracy, addressing the issue of uncertainty estimation in VLM predictions.
• Performing further experiments using MM-CoT to evaluate the impact of prompting and reasoning on performance, particularly in comparison to closed-source models like AHA.

We hope these revisions comprehensively address your concerns. If so, we would greatly appreciate it if you could consider raising our scores.

Thank you once again for your valuable feedback and consideration.

We thank the reviewer for their thoughtful comments and for engaging with our explanation. To clarify, we do not claim that the likelihood is perfectly correlated with accuracy (cosine similarity) or always correlated. Instead, we argue that there is a general trend where higher likelihood is associated with higher accuracy. This trend is observed in AHA, where predictions with higher likelihood tend to exhibit higher accuracy. Conversely, the baseline model demonstrates the opposite behavior, with lower likelihood correlating with lower accuracy.

We acknowledge the importance of quantitatively evaluating this relationship and agree that metrics like Spearman correlation could provide a rigorous way to assess the alignment between likelihood and accuracy. We will consider incorporating such analyses to strengthen our evaluation and further illustrate the differences between AHA and the baseline model.

Thank you for the explanation. If I understand correctly, you use the likelihood as the uncertainty/confidence measure. The desired relationship between the likelihood and the accuracy (similarity) should be linearly proportional, assessing how well the predicted uncertainty coincides with the prediction error, i.e., the higher the likelihood, the higher the accuracy, and vice versa. One example can be the Area Under Sparsification Error (AUSE) [1]. The current results don't express such a correlation as the underperformed model might have a higher correlation.

[1] Lind, Simon Kristoffersson, et al. "Uncertainty quantification metrics for deep regression." Pattern Recognition Letters 186 (2024): 91-97.

We appreciate the reviewer’s feedback and apologize for any confusion. For uncertainty estimation experiments, we used two metrics:

1. Cosine Similarity: This measures the similarity between the ground-truth sentence and the model’s prediction by calculating the cosine of the angle between their vector representations. These vectors are typically derived using embedding methods such as TF-IDF, Word2Vec, GloVe, or modern techniques like BERT. Which the closer to 1, the cosine similarity the more accuracy is the model prediction. (Which was also used in the main paper as one of the key metric of evaluation, in Table 2)

2. Sentence Token Likelihood: This metric evaluates the probability of a sequence of tokens forming a sentence, as assigned by a language model. It reflects the model's confidence in its predictions. We compute this by first finding the model's individual token probability, and the overall likelihood of the sentence is calculated by multiplying the probabilities of all the tokens in the sequence. The closer the score to 1, means the more confident the model is confidence about its prediction of the sentence.

We use Cosine similarity as a way of measuring semantic meaning accuracy of the model in the paper, and in this additional experiment we introduce sentence token likelihood to measure confidence of the model's prediction. Hence, our results show that AHA outperforms LLaVA-13B-v1.5 in generating language outputs more similar to the ground truth (higher cosine similarity) while also exhibiting greater confidence (higher sentence likelihood). This demonstrates the positive impact of fine-tuning with the AHA dataset. We hope this clarifies our approach. Thank you.