HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging

Q1. Can the authors provide the prompts used in the method?

Thank you for the suggestion. We have added an additional section (A.8) in the appendix of the revised version to provide some prompts used in our method. Please note, we have already provided most of the prompts in the Supplementary Material. We will open-source all the code and framework if the paper gets accepted, and the prompts will be included as well.

Q2. Were repeated experiments conducted in the experimental section to mitigate the impact of randomness on model repair?

Thank you for the reminder. In the revised version, we have updated Table 2 with the average results from five repeated runs using different random seeds. The overall performance remains largely consistent across these runs.

Q3. Is it possible to verify whether the error slices from different clusters correspond to distinct bugs in the model?

Thank you for this insightful question; it addresses an important aspect of error slice discovery and model debugging. We have added this interesting experiment in the appendix, section A.1.6, of the revised version.

In our paper, error slices are defined based on the combination of tags, and therefore, slices may overlap in some tags. While error slices are distinct in terms of their tag combinations, they do not necessarily correspond to distinct bugs. Fixing one error slice may improve the model’s performance on other related error slices as well.

To test this, we designed a simple experiment to check the impact of fixing one error slice on others. For the ResNet model used in the classification task described in the main paper, we chose an error slice corresponding to class teddy bear, defined as (object color: white, object pose: sitting). We then queried the data matching this error slice (20 images) from a hold-out dataset. We fine-tuned the model on this data for 20 epochs and evaluated its accuracy on the validation set. We collected the model’s accuracy on three types of data:

1. Data matching the selected error slice.
2. Data belonging to error slices that overlap with the selected slice (e.g., (object color: white, xxx), (xxx, object pose: sitting)).
3. Data belonging to error slices that does not overlap with the selected slice.

The results reveal a significant improvement in the model’s performance on the fixed error slice. Interestingly, performance on overlapping error slices also improved, suggesting a potential relationship between these slices and shared model weaknesses. However, performance on non-overlapping data decreased slightly, likely due to overfitting to the specific distribution of the fine-tuning data. This experiment suggests that while error slices are defined by distinct tag combinations, they are not distinct bugs.

Q1: What is the naive approach to slice enumeration?

Thank you for pointing this out. This algorithm refers to the brute-force method mentioned in Section 4.1.2. It simply lists all possible data slices and then searches for matching data for each slice. We have now added this explanation in the revised version of the main text (section 5.2) and included an algorithm description in the appendix, section A.7, for further clarification.

Q2: How is the tree-structured baseline approach different from DebugAgent's method?

Thank you for raising this point. Compared with DebugAgent, this approach lacks the pruning and the Intersection discussed in Section 4.1.4. Compared to the naive approach, the tree-structured baseline method progressively increases the number of attributes included in each data slice during the search. For example, when searching for all possible data slices for a combination of three attributes, the algorithm first searches for slices with one attribute, then with two attributes, and finally with three attributes. When searching for matches for a three-attribute combination, the search is restricted to the matching data from the parent node, improving search efficiency. We have now added this explanation in the revised version of the main text (section 5.2) and included an algorithm description in the appendix, section A.7.

Q3: How difficult is comparing tag generation time between DebugAgent and the SoTA? (At least HiBug).

Thank you for your insightful question. Comparing the tag generation time between DebugAgent and SoTA methods such as HiBug is challenging due to the different factors that influence time consumption. For HiBug, the primary factor is GPU capacity, while for DebugAgent, it is network bandwidth.

The most time-consuming part of attribute and tag generation (also of the whole method) is the tag assignment for all the images in the dataset, which typically accounts for over 95% of the total time, with larger datasets increasing this proportion. HiBug uses the BLIP VQA model, whereas our method uses GPT-4V. In practice, for HiBug, generating labels for a single attribute on one image takes around 2 seconds (using a 4060 GPU), and generating labels for all attributes for that image takes about 2×M seconds (where M is the number of attributes). For DebugAgent, generating labels for all attributes for one image takes around 6 seconds (using the GPT-4V API). If we consider generating labels for each image individually, HiBug’s time complexity is O(N×M), where N is the total number of images and M is the number of attributes. Our method, however, operates with O(N) because GPT-4V can generate all attributes in one step.

In practice, parallel processing is typically used to accelerate these processes. With unlimited GPUs, HiBug could label all data simultaneously, and similarly, with unlimited threads and network bandwidth, DebugAgent could generate labels for all data in 6 seconds. Due to the different influencing factors, it is difficult to establish a fair and standardized comparison.

Thanks again for pointing out this issue. Time cost of DebugAgent is indeed an aspect that we did not detail in the paper. We have now added a discussion of the time cost of DebugAgent to the revised version of appendix, section A.1.5.

Q4: Is it possible to identify slices from other SoTA tag generators and compute their performances with the same models?

Yes, it is possible. Our slice enumeration algorithm can be directly applied to HiBug. Additionally, our method can be used on datasets that already have attributes and tags, such as the CelebA dataset, as long as all images in the dataset are labeled with a consistent set of attributes. However, our method cannot be applied to general tag generators, such as Recognize Anything[1], because these methods do not label images with a consistent set of attributes, which hinders the implementation of slice enumeration.

Q5: Is it possible to provide the shape (number of instances, features, and unique labels) of the private dataset used for pose estimation or an alternative proxy public dataset?

The pose dataset contains 47,057 images (24,832 images meticulously curated from COCO, the remainder from a private source), primarily used for rehabilitation training in hospitals to recognize patient movements and assess whether the designed exercises meet the required standards. In our experiments, models are pre-trained on the COCO portion. For error slice discovery, we use 7,057 images from the private portion as the validation set. For model repair, we select 1,241 images from the private portion. We have included these information in the appendix, section A.3, of the revised version. We have also presented examples of error slices from this dataset (with faces blurred for privacy) in the appendix, section A.4, of the revised version.

W1. The formatting of Figures 1 and 4 lacks aesthetic appeal; for Tables 1 and 2, it is recommended to utilize triple-line tables.

Thank you for the suggestion. In the revised version, we have updated Tables 1 and 2 to use triple-line tables for improved clarity and presentation. We have also updated Figure 4 to enhance its aesthetic appeal. For Figure 1, its current design prioritizes simplicity and clarity for conveying the information effectively. Adding additional elements, such as colors, may compromise its readability and detract from its intended purpose.

W2. In the experimental section, the efficacy of attribute and label generation, as well as slice enumeration, was individually assessed in Part V, without comprehensive comparisons to existing methodologies.

Thank you for your comment. We would like to clarify that we have compared the efficacy of attribute and label generation to existing methods, as shown in section 5.1 and appendix A.2.

As for slice enumeration, since the tag-then-slice approach is a relatively recent development, this issue has not been specifically addressed by previous methods. Specifically, HiBug focuses on error slice discovery based on a single attribute tag and does not consider attribute combinations. However, we found a similar slice discovery algorithm in HiBug's code, which has a structure similar to the naive approach used in our baseline method. Given the limited number of existing methods for comparison, we only include comparisons with custom-built baseline algorithms.

W3. In the abstract, it is mentioned that DebugAgent visualizes the attributes of task generation through an interpretable and structured process. While this structured approach is referenced elsewhere in the text, its specific meaning in the context of utilizing GPT for attribute and label generation in Section 3.2 may not be entirely clear.

Thank you for your comment. We understand that the term "visualizes" in the review may have been misunderstood. In the abstract, we state that DebugAgent generates task-specific visual attributes through an "interpretable and structured process."

The "interpretable" aspect refers to two key points: first, our approach follows a clear and logical structure; and second, the generated attributes and tags are human-understandable. The "structured" aspect refers to our systematic method for prompting GPT to generate relevant attributes. Specifically, we first define three types of visual attributes and two sources of error. Then, we extract the visual attributes based on the intrinsic difficulty of the task and the characteristics of the dataset.

W4. In Section 4.2, the mention of tag substitution and instruction-based method is aimed at predicting erroneous segments beyond the validation set. While Table 1 illustrates model degradation in predicting erroneous slices using these methods, Section 5.4 reveals that the model's average performance on these predicted erroneous slices significantly falls below its overall performance, emphasizing the efficacy of the DebugAgent approach in predicting additional erroneous segments. Your query highlights a potential inconsistency: the absence of testing the effectiveness of the DebugAgent approach in predicting additional erroneous segments, coupled with the fact that tag substitution and instruction-based method are proposed by your team, revealing a logical gap in the narrative.

If we understand this comment correctly, the concern is the lack of baselines for evaluating our methods of predicting unseen slices. We clarify that:

1. Predicting erroneous slices is a novel aspect of our work that rarely attempted in previous error slice discovery methods, which is why there is no direct baseline comparison.
2. The primary function of DebugAgent is to systematically identify and repair erroneous slices based on the dataset validation set. In contrast, tag substitution and instruction-based methods are auxiliary tasks used to hypothesize additional erroneous slices that may not be covered by the validation set.

We have clarified this distinction in the revised version to emphasize that the lack of baseline comparison stems from the novelty of our approach.

W5. Some data points are referenced in Section 5.3 without accompanying tables for display.

Thank you for your comment. The data points referenced in Section 5.3 refer to the number of error slices identified for each dataset and model. However, since the number of models used varies across datasets, it would be difficult to present this information in a single, unified table.

Dear Reviewer KQpB,

Thank you again for your feedback and for engaging with our work. We wanted to kindly follow up regarding the discussion. While the discussion period remains open for another week, tomorrow marks the final deadline for any further revisions to the paper.

If there are any additional concerns, experiments, or clarifications that you feel are necessary, we would be happy to address them promptly within the discussion. Please let us know if there is anything we can do to further improve the paper or clarify our responses.

Thank you once again for your time and effort.