Dear R-ctad,

Thank you for the feedback which helped to clarify our paper. We'll address misunderstandings about contextual information and LLM necessity.

We provide grouped answers A-D & highlight paper updates.

---

(A) Clarification on contextual information

We apologize for any confusion caused by our use of the term 'context'. To clarify, we use 'context' in two distinct ways: (a) external input by a human (not strictly necessary for the framework) and (b) external knowledge (such as relevant background knowledge, domain understanding) within an external system (LLM). While we made a small footnote about this on page 5, we agree this is not enough and could have resulted in confusion about the nature of our work.

"Context-aware testing" refers to external knowledge use, not user input. SMART uses tabular context (which is inherently encoded in the feature names) for hypothesis generation without needing human input. For example, Table 1 in the paper showcases the hypotheses that were generated by SMART leveraging the feature names without any external human effort.

Need for prior knowledge & practical example.

Further, based on your suggestion to include a 'practical' example to illustrate the application of the proposed method, we thought it would be useful to show how easy it is to use SMART as well as the minimal contextual information and dataset contextualization needed.

As shown below, users do not need any prior knowledge to use SMART. Rather, we make use of the context inherently encoded in the dataset, feature names and task.

import SMART
model_tester = SMART('gpt-4')

# Optional external input (can be left blank)
context = "Find where a model fails. This is a prostate cancer prediction task."

model = XGBoost()
description_dataset = X_train.describe()

model_tester.fit(X_train, context, description_dataset, model)

print(model_tester.subgroups)

This implements SMART with just a few lines, requiring no additional human effort beyond providing the dataset and model. In our implementations, we typically provide generic, easily accessible context that does not vary across tasks (see Appendix C.3. for examples).

UPDATE: Add table to Sec. 3.3 to clarify the two uses of context and rename the context input as external input which we hope will disambiguate the two meanings.

---

(B) Difference from Rule-Based Methods

CAT and SMART fundamentally differ from rule-based methods:

• No human input required: Unlike rule-based methods, CAT does not need experts to define rules based on domain knowledge.
• Automated hypothesis generation: SMART uses feature names as implicit context to guide hypothesis generation automatically.
• Scalability: This approach can be applied to any tabular dataset without human intervention.

UPDATE: Add a contrast to rule-based in Sec. 3.2

---

(C) Why we need an LLM in SMART

We wish to avoid using humans for large-scale testing because human intervention is expensive, time-consuming and often unavailable. Instead, we desire an automated approach to testing ML models. For this, we suggest to employ LLMs. The use of an LLM is crucial because (as per L177-186):

• Contextual understanding: LLMs can interpret feature names and generate relevant hypotheses without human input
• Targeted sampling: LLMs limit the number of tests, focusing on relevant model failures without testing all possible combinations

Benefits. We show that by using LLMs to search for model failures, we are able to test ML models better. SMART finds data slices where models are unreliable much more often than data-only methods (see Sec. 5). Therefore, SMART offers a solution when human experts are unavailable or too expensive.

Ultimately, by using LLMs to automatically generate context-aware hypotheses, we can test models for model failures without succumbing to the issues that data-only testing methods face (see Sec. 3.2).

UPDATE: Enhance current description around L174-186.

---

(D) Questions

Q1: What constitutes contextual info?

A1: See response (A) clarifying contextual information.

Q2: How are the synthetic features provided in SMART testing in Sec. 5.1?

A2: We add new random features to the dataset which have uninformative names, such as "feature1", "feature2". We show that data-only methods will test for everything which results in systematic errors. SMART avoids this by selectively (and automatically) testing for important data slices.

Q3: How is the method applicable if there are no descriptions of features?

A3: See response (C): (a) feature names provide sufficient context for hypothesis generation; (b) generic prompts are used that do not change across tasks (see Appendix C.3 for examples). There is no need to provide additional descriptions.

Q4: How are steps (1) and (2) in SMART beneficial if we already have contextual information about age and comorbidities?

A4: They are beneficial because we do not assume prior access to any knowledge (see response (A)). These hypotheses are automatically generated by the system purely from tabular feature names & generic external input about the task.

---

Thank you

Given our clarifications, we hope you consider revising the paper's evaluation ☺️

Dear reviewer ctad,

We are sincerely grateful for your time and energy in the review process.

We hope that our responses have been helpful. Given the limited time left in the discussion period, please let us know if there are any leftover concerns and if there was anything else we could do to address any further questions or comments :)

Thank you!

Paper authors

Thanks for your continued engagement. We're glad we've addressed prior concerns and reached a common understanding on some points. We clarify the remaining comments, which we hope persuades you to increase your score.

a) Should we be using SMART without clear feature names?

We're glad that we're in agreement that we also do not recommend using context-aware testing, and SMART specifically without feature names. While we see this reliance on the contextual nature of feature names as a benefit (by the nature of context-aware testing), we acknowledge we can be more explicit about this. We will expand the discussion section that data-only methods should still be the go-to testing approach in the rare cases in practice where feature names are uninformative.

b) What about when feature names change?

Regarding your OpenML benchmark example, SMART can handle variations in feature names as long as they remain informative (e.g., "age" vs "patient_age"). We have consistently demonstrated in the paper that SMART can generate meaningful hypotheses with diverse column names. Thus, changing the name would not affect the reliability. That said, we have another safety mechanism in place --- our self-falsification mechanism (L209-224) which uses empirical data to evaluate each hypothesis. Finally, to clarify, in real applications in both research and industry, we find that the models are tested against a single dataset version containing informative feature names.

c) Is this a critical limitation in deep learning tabular problems?

You also mention that this is a limitation with "LLMs in tabular deep learning problems". This is the first time you have raised this concern. However, this critique is simply not true. As discussed, both our framework context-aware testing and our method SMART specifically are model-agnostic (discussed in L140-162). We give both theoretical and empirical reasons for this.

• We show theoretically why our framework is model agnostic. We develop context-aware testing because we show that the alternative --- data-only testing--- implictly falls into the 'multiple testing problem' (Sec. 3.1.). This issue of ML testing does not depend on the underlying model. We address these theoretical issues of data-only testing methods (Sec. 3.2.) even in deep learning models.
• We provide empirical results to support this claim with deep learning methods. In addition to our original evaluation which included MLPs (Table 12 - main paper), we have updated with two new deep learning tabular models (Table 4 - response pdf).

d) What is the novelty of our study?

We're quite surprised to see your comment that our novelty is limited.

In your 'summary' section of the original review, you summarized the following (taken as quotes):

• Unlike previous methods that rely solely on data to find slices where a model's predictions underperform compared to average performance, CAT addresses the limitations of these data-only approaches.
• Traditional methods often fail to consider the problem context and can lead to multiple testing problems.
• To overcome these issues, the authors propose loosening the restrictive assumption of relying solely on data

In addition, you highlighted our contributions in the strengths:

• 'the core limitation of data-only testing—the lack of a principled failure mode discovery procedure that can incorporate prior knowledge to guide the search for meaningful errors—is also persuasive.'

We are concerned that there may be some misalignment between your original review and your current comments.

To wrap up the discussion around our contributions, it seems that you acknowledge that we:

• Identify a fundamental limitation in testing methods which rely only on data (Sec. 3.1 in the paper).
• Provide a theoretical mechanism explaining why data-only testing falls short, relating it to the foundational view in statistics of multiple model testing in frequentist statistics. (Sec. 3.2)
• Proposing a new paradigm which avoids the theoretical issues: selectively testing the most relevant tests instead of testing for everything --- and falling into the multiple hypothesis testing problem again; (Sec. 3.3)
• Building a concrete system - SMART which is an auditable, easy-to-use, automated system as an alternative to data-only testing (Sec. 4)
• Showing that this works empirically on over 12 quantitative & 5 qualitative evaluations. (Sec. 5)

Therefore, we respectfully disagree with your most recent comment that our work "merely introduces an LLM to identify model failures" --- which also overlooks our clarification in our previous Answer 3

e) Finding consensus.

We hope our responses and clarifications addressed your main questions and concerns? If so, we kindly ask that you consider increasing your score to better reflect this. If not, please let us know what specific unadressed issues remain, and we'll do our utmost to resolve them.

Dear R-H6db,

Thank you for your thoughtful comments to improve the paper. We provide answers (A)-(E) & highlight updates to the paper

---

(A) Clarifying contextual information

We'd like to clarify that when we refer to "context", we mean the relevant background knowledge of a system (e.g. an LLM) that informs the generation of targeted hypotheses.

(i) Dependence on context: To clarify, we don't require external contextual information or human expert input. For tabular data, context is inherently encoded in the feature names. Our focus on tabular data leverages this built-in context. For instance, in a medical dataset, features like age, sex, or patient features provide context to guide LLM hypothesis generation. This contrasts with data-only approaches which only use the numerical data values alone and ignore the context surrounding the feature names.

(ii) Identifying relevant context: We do not require manual selection of contextual information. The relevance or value of the contextual information is automatically determined by the LLM: (a) when proposing hypotheses based on tabular data's inherent context and (b) via our self-falsification mechanism which empirically evaluates hypotheses and prunes those not supported by data.

UPDATE: We will update Sec. 4 using the extra camera ready page to clarify this concept of context and how it is used.

Finally, to highlight the minimal input needed, we thought it would be useful to show SMART's ease of use. i.e. additional contextual info input is not necessary but possible. If not provided the LLM leverages the context inherently encoded in the dataset X.

import SMART

# Instantiate SMART
model_tester = SMART('gpt-4') 

# Give desired context (this can be left as an empty string)
context = """Find where a model fails. This is a prostate cancer prediction task."""

# Load ML model
model = XGBoost()
dataset_description = X.describe()

# Test the model
model_tester.fit(X, context, dataset_description, model) 

UPDATE: include the code snippet in the revision to show usage of SMART.

---

(B) Alternatives to the LLM for hypothesis generation

To the best of our knowledge, the current main alternative for generating targeted hypotheses would be human input. However, this lacks SMART's automation and scalability. This underscores both the novelty of our LLM-based approach for targeted model testing. Additionally, this highlights exciting research opportunities to develop alternative targeted samplers for hypothesis generation within the proposed CAT framework.

--

(C) LLM bias & mitigation strategies

We agree with you that is important to look at directions for removing biases from SMART. Our discussion on precisely this issue can be found in Sec 5.4. We realize that the title of the section might not fully convey this and hence will rename it (see update below).

To summarize, in Sec 5.4 we have outlined several strategies to mitigate biases inherent in LLMs.

• (a) Using the training dataset to guide hypothesis generation.
• (b) Use our self-falsification mechanism to empirically test hypotheses against the actual data, thereby filtering out spurious hypotheses.
• (c) Transparent Reporting: SMART generates comprehensive model reports (example in Appendix D.7) that provide clear justifications for each hypothesis, allowing for human oversight and intervention if needed.

Empirically, Sec. 5.4 (Table 4) demonstrates SMART's ability to correctly identify underperforming subgroups, even in scenarios where LLMs might have prior biases (e.g., ethnicity-related biases).

UPDATE: Rename section 5.4 as “Assessing and mitigating potential LLM challenges and biases” to better flag this.

---

(D) CAT beyond tabular data - images

Our focus on tabular data is intentional due to its importance in many high-stakes domains such as healthcare, finance, and criminal justice, making ML testing particularly impactful for these critical applications by addressing a significant gap in current data-only ML testing methodologies.

Additionally, tabular data inherently includes context through feature names and metadata. This context is not naturally present in images (tensor of pixel intensities). As discussed in Appendix A, extending CAT to other domains like images would require incorporating metadata to provide necessary context — which is often unavailable.

While extending CAT to other domains is an interesting future direction, it's non-trivial. We hope future work can build on our work and adapt the context-aware paradigm to images.

UPDATE: add this discussion to our discussion in Appendix A.

---

(E) Constraint on LLM: might not be accessible or applicable in all scenarios

We wish to clarify the value of SMART even with possible constraints. First, LLMs are becoming increasingly accessible, especially from a cost perspective. As shown in Table 1 (Response pdf) it costs less than 0.10 USD for 5 hypotheses and less than 0.50 USD for 100 hypotheses for state-of-the-art LLMs.

Second, we show that SMART outperforms data-only methods even when using cheaper, less capable LLMs like GPT-3.5 (Appendix D.3) which are widely available to the public.

Third, research done within the ML community is not always accessible or applicable (as illustrated by the vast research agenda of deep learning which presupposes access to expensive GPUs). Similarly, we see our research as being important in pushing the frontier of ML testing despite presupossing access to an LLM.

Finally, given the stark differences in the effectiveness of data-only and context-aware methods (refer to the global response table), we think it is extremely important for the research community to have access to improved testing methods for more reliable ML testing in high-stakes domains.

---

Thank you

Thank you for your engagement. Given these changes, we hope you might consider revising your assessment of the paper's impact.

Dear reviewer H6db

We are sincerely grateful for your time and energy in the review process.

We hope that our responses have been helpful. Given the limited time left in the discussion period, please let us know if there are any leftover concerns and if there was anything else we could do to address any further questions or comments :)

Thank you!

Paper authors

Thank you for your clarification. After reviewing your answer (A), I remain unclear about how the context information is generated. You mentioned that 'context is inherently encoded in the feature names' and that no external contextual information or human input is required. Does this imply that the LLM can automatically infer context purely from feature names?

Consider two identical tabular datasets with the same feature names—such as name, age, gender, occupation, and income—but with different prediction objectives: one for deciding bank loans and the other for predicting job promotions within five years. Both have binary outputs (yes or no). How would the LLM infer different context in these two different scenarios?

import SMART

# Instantiate SMART
model_tester = SMART('gpt-4') 

# Give desired context <--- need to specify this in your example
context = """Find where a model fails. This is a prediction task about whether to give bank clients bank loan or not."""

# Load ML model
model = XGBoost()
dataset_description = X.describe()

# Test the model
model_tester.fit(X, context, dataset_description, model) 

Dear reviewer H6db,

Thank you for the engaging exchange we had that helped to clarify our work. As the discussion period is coming to a close, we'd like to follow up on your prior concerns.

Have our responses and clarifications addressed your main questions and concerns? If so, we kindly ask that you consider increasing your score from a 5 to better reflect this. If not, please let us know what specific issues remain, and we'll promptly provide additional information to resolve them.

We appreciate your reconsideration and look forward to your feedback.

Dear reviewer bfyv,

Thank you for taking the time to carefully review our work. We provide answers below (A-F) and highlight our paper updates.

---

(A) Method degradation with less informative context

We appreciate your interest in SMART's performance under varying context quality. We've developed the context-aware framework primarily for the tabular domain because feature names encode useful information for reasoning about potential failures. Therefore, we do not recommend using CAT when feature names do not encode useful information.

We address your concern in two ways. First, we perform a qualitative study where we limit the data schema by hiding the feature names and inspect the hypotheses and justifications generated. We find that in the limited-schema case, SMART generates hypotheses based on inferences about the feature information (e.g. "the model might fail on feature_4 if feature_4 represents gender"). In contrast, informative names guide meaningful hypothesis generation. Such hypotheses and justifications are illustrated in Table 3 in the response pdf.

Second, we evaluate whether limiting the data schema by hiding some feature names and leaving minimal external context affects detection rates of model failures. We compare two versions of SMART, original and with corrupted feature labels, in identifying data slices with high performance discrepancies from average (Fig. 1 in the response pdf). We find that across two real-world private datasets, hiding the feature names hinders model evaluation. This highlights that feature names play an important role in finding model failures.

UPDATE: We'll discuss SMART's sensitivity to feature names in the appendix.

---

(B) Moving content from the appendix to the main text

We agree with you that highlighting the specific prompt structure used would improve readability and we'll add more discussion in the camera-ready version.

We will also clarify our data subset selection methods (slices are always selected based on train data and evaluated on test data).

---

(C) Related work

We appreciate you bringing attention to relevant related work. Both works [1] and [2] differ from our approach in that they focus on data validation techniques for general ML pipelines, while our context-aware testing framework specifically aims to use context to guide the search for model failures.

UPDATE: We will cite these works in the revision and provide a discussion of how context-aware testing differs from existing data validation techniques used in industry.

---

(D) Comparison on TableShift benchmark

While TableShift [3] is a useful benchmark, it's not directly applicable to SMART. TableShift primarily focuses on OOD-detection. In contrast, SMART searches for model failures within the existing data distribution (recall that the context-guided sampling mechanism in Def. 1 samples from the existing data distribution).

That said, we agree that addressing dataset shift would be a valuable extension of context-aware testing. We'll mention this as future work, highlighting TableShift as a potential benchmark for extending context-aware testing to the OOD domain.

---

(E) Code release

We expect to release the code in the form of a package mirroring your suggestion. We demonstrate the usage capabilities here and will expand it in the appendix.

import SMART

# Load SMART
model_tester = SMART('gpt-4') # Instantiates smart

# Give desired context
context = """Find where a model fails. This is a prostate cancer prediction task."""

# Load a desired ML model
model = XGBoost()

# Test the model
model_tester.fit(X_train, context, model) 

Once fitted, many attributes can be accessed, such as found subgroups model_tester.subgroups, generated hypotheses model_tester.hypotheses, justifications model_tester.justifications, model report model_tester.model_report, and more. We can also evaluate it: print(model_tester.top_underperforming_subgroup)

Overall accuracy of the model: 85.1%
Accuracy on 'Age > 75' subgroup: 55.2%
Discrepancy of: 29.9%

UPDATE: Add code usage to appendix.

---

(F) Open-weight models for hypothesis generation

Before jumping in, we wish to highlight Appendix D.3, which notes our goal is not to exhaustively test the SMART framework with every LLM. Rather, the goal is to showcase that SMART is feasible with at least the capabilities of GPT-4. While we assessed SMART's performance with different LLMs, we caution against using it with less capable models.

That said, based on your comment we have conducted an experiment with Mistral-7b, Qwen-1.5-7b, Llama-3-8b, Llama-70b, where for the OULAD and SEER datasets we generate 5 hypotheses and assess overlap to the hypotheses generated by GPT-4. This is presented in Table 2 in the response pdf.

To summarize, the overlap between open-source models and GPT-4 is between 60-80%. We find that open-source models propose similar hypotheses, but they are not replacements for more capable models. This corresponds to our findings in Appendix D.6 --- less capable models might propose similar hypotheses, yet they still catch fewer model failures.

Closed-source models are cheap. In case your worry is that closed-source models are expensive to run, we conduct an additional evaluation calculating the cost of running SMART with closed-source models (Table 1 in the response pdf). The cost does not scale with dataset size but with the number of tests conducted. We find that generating 100 hypotheses with a powerful closed-source model (GPT-4o-mini) costs less than 1 cent and using state-of-the-art models (GPT-4) costs about 0.30 USD.

---

Thank you

Thank you for your engagement. Given these changes, we hope you might consider revising your assessment of the paper's impact.

Dear R-6t3M,

Thank you for your thoughtful comments to improve the paper. We provide answers (A)-(F) & highlight updates to the paper.

---

(A) Highlighting our systematic comparison of SMART vs data-only

We appreciate your suggestions and believe there may be a misunderstanding regarding the extent of our comparisons. We wish to clarify we provide in total 11 quantitative and 2 qualitative experiments (see global response table above), as well as five new additional insight experiments in the response pdf.

UPDATE: To help clarify, we add a new summary table (Global response table) with the main takeaway and references to the relevant sections, tables and figures.

We hope this addresses your concerns and demonstrates the thoroughness of our comparisons.

---

(B) Clarifying tabular models used

We thank you for your comment on the choice of downstream tabular models. We clarify that a core contribution of SMART is its targeted sampling of hypotheses, which is entirely independent of the model used. As detailed in Section 3.3 and Section 4, SMART's context-guided slice sampling mechanism $\pi$ is used to generate hypotheses independently of the downstream model. This is why the specific tabular model used is not the primary focus of our paper.

Despite this, we demonstrate SMART's effectiveness across multiple models, including Logistic Regression, SVM, XGBoost, and MLP, showing the applicability of our findings.

While we understand the reviewer's concern about "outdated" models, we argue that our chosen models remain highly relevant for tabular data:

• Interpretability: Models like logistic regression offer crucial interpretability in domains such as healthcare and finance and are still widely used in industry.
• Performance: Recent work has shown that traditional ML models like tree-based boosting methods (XGBoost) often outperform deep learning approaches on typical tabular data [Grinsztajn et al., NeurIPS 2022] and we would argue that tabular deep learning has not been universally adopted for tabular data.

That said, we have also run additional results with two tabular deep learning method: TabPFN and TabNet (Table 4 in the response pdf). Across all the methods, SMART is the best at finding subgroups where the models are least reliable.

---

(C) Cost of LLM hypothesis generation & scalability to larger datasets

We clarify the cost and scalability of SMART — demonstrating not only that SMART is cheap but also easily scalable to large datasets

• Scalability: SMART's scalability depends on the number of hypotheses generated, not dataset size (unlike data-only methods). This allows SMART to easily scale to arbitrarily large datasets.
• Cost Analysis: In practical terms, cost then also scales primarily with the number of hypotheses generated, not dataset size. We provide a rough estimate based on token counts of input and outputs for 2 datasets (SEER and OULAD). This would be $<0.10$ USD for 5 hypotheses and $<0.50$ USD for 100 hypotheses for state-of-the-art models.

---

(D) SMART beyond tabular data (e.g. NLP)

We appreciate your interest in the generalizability of SMART. Our current focus is on tabular data due to its importance in high-stakes domains like healthcare and finance, as well as, context being naturally encoded in tabular data which is used to guide the targeted sampling of failures. Extending SMART to other domains like NLP requires non-trivial adaptation, warranting dedicated investigation.

Applying SMART to text would require addressing the lack of explicitly interpretable features and developing new ways to operationalize hypotheses on unstructured data. In contrast, tabular data offers inherent context due to the interpretability of features, which is crucial for context-aware hypothesis generation and operationalization.

UPDATE: We'll use the extra camera-ready page to highlight this as an interesting future avenue for extending the context-aware testing paradigm.

---

(E) Discussion on future bias mitigation when using SMART

We agree with you that is important to look at directions for removing biases from SMART. Our discussion on precisely this issue can be found in Sec 5.4. We realize that the title of the section might not fully convey this and hence will rename it (see update below).

To summarize, in Sec 5.4 we have outlined several strategies to mitigate biases inherent in LLMs.

• (a) Using the training dataset to guide hypothesis generation.
• (b) Use our self-falsification mechanism to empirically test hypotheses against the actual data, thereby filtering out spurious hypotheses.
• (c) Transparent Reporting: SMART generates comprehensive model reports (example in Appendix D.7) that provide clear justifications for each hypothesis, allowing for human oversight and intervention if needed.

UPDATE: Rename section 5.4 as “Assessing and mitigating potential LLM challenges and biases” to better flag this.

---

Thank you

Thank you for your engagement --- we believe these changes should improve the paper's contribution and clarity. Given these changes, we hope you might consider revising your assessment of the paper's impact and the corresponding evaluation for NeurIPS.

Dear reviewer 6t3M

We are sincerely grateful for your time and energy in the review process.

We hope that our responses have been helpful. Given the limited time left in the discussion period, please let us know if there are any leftover concerns and if there was anything else we could do to address any further questions or comments.

Thank you!

Paper authors