TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring

Q4: Comparison of data distributions between the authors’ dataset and related work

We have provided a table summarizing how TrustSQL differs from other benchmarks in GC1 of the general comments section.

Q5: Definition of query familiarity and query difficulty

Query familiarity refers to whether a question's original template is included in the training set. Unlike crowdsourced datasets that use template-free data creation, TrustSQL leverages a template-based approach. In the SQL query validation process (detailed in our re-annotation process), we verify that question templates are semantically the same. Given that there are over 180 templates on average per each dataset, assessing pairwise similarity directly is impractical. Therefore, we adopted a two-stage procedure: (1) We grouped question templates that use the same placeholders. For example, "Tell me flights from city_name1 to city_name0" and "What are airlines that provide services from city_name1 to city_name0" share the placeholders city_name1 and city_name0; (2) Within each group, we reviewed the templates to determine if they are semantically identical. If they were, we merged the templates and paired them with a unique SQL structure. This process allows us to maintain control over the semantic distribution of questions across data splits.

Regarding the classification of query difficulty, we acknowledge that SQL complexity alone does not encompass all challenges in text-to-SQL tasks. Nonetheless, we believe SQL complexity remains the most significant factor in assessing query difficulty, as it directly reflects the complexity of the model's expected output. While BIRD, as the reviewer mentioned, introduces four dimensions to measure sample difficulty, these guidelines are inherently subjective and may vary among annotators. For instance, their annotation framework rates 1 as a simple SQL with few keywords, 2 as more complex than 1, and 3 as a highly complex SQL with many functions, which can result in inconsistencies across samples. Similarly, while DIN-SQL's approach is not entirely rigorous, it is straightforward and demonstrates strong correlations with model performance—as query difficulty increases, performance decreases (see Appendix B.2 on SQL generation).

Q6: Diversity and distribution of questions

We utilize existing dataset question templates, so the types and distribution of templates reflect those in the original datasets. The number of question templates for ATIS, Advising, and EHRSQL is 238, 141, and 168, respectively. In our paper, the term "diverse input" refers to a setting where the text-to-SQL model is tasked to process various types of questions, including both feasible and infeasible ones.

Q7: The selection of c

We have elaborated on the selection of c in the general comments section above (see GC4).

Q8: The choice of fine-tuned models (SQLCoder-2)

SQLCoder-2 (defog/sqlcoder-7b-2) is an open-source SQL-specialized model based on CodeLlama 7B. Although it may not be widely used in cross-domain generalization benchmarks, this was not a key consideration, as we do not conduct cross-domain tasks in our experiments. SQLCoder-2 was chosen because it outperformed the fine-tuned Llama 3.1 8B in our initial tests. We have clarified in the manuscript that SQLCoder-2 is one option for a decoder-only model, whereas T5-3B is an encoder-decoder model.

Q9: Compatibility of abstention mechanisms across fine-tuned models

SQLCoder-2 is a decoder-only model, while T5 is an encoder-decoder model. Their respective abstention methods may not always be compatible. We chose these to demonstrate how different architectures can be paired with compatible abstention mechanisms in various ways.

Thank you for your detailed comments on our paper, which have been very helpful in improving the clarity of our manuscript. We have restructured your questions and addressed them point by point.

Q1: Domain generalization concerns and the reason for not using Spider and BIRD

The inclusion of the three domain-specific databases aims to evaluate whether methodological trends for reliable text-to-SQL modeling remain consistent across different databases, rather than assessing a model's cross-domain generalization ability in SQL generation. Our benchmark focuses on evaluating a model's ability to discern whether to generate SQL or abstain, emphasizing reliability over general SQL generation across varied databases. Unlike benchmarks aimed at domain generalization, our task prioritizes reliability in domain-specific challenges—handling questions that reflect user needs, complex databases, and rich question-SQL assumptions—while incorporating penalties for incorrect decisions. Consequently, high-quality annotations are essential to accurately assess model performance in these specific contexts. To ensure this quality, we ruled out large-scale crowdsourcing and manually curated annotations using three complex domain-specific datasets to maintain the benchmark's integrity.

We also considered using only the development sets of Spider and BIRD but excluded them for the following reasons. Spider is nearly solved as a benchmark, with most errors arising from annotation issues rather than modeling challenges, which limits its usefulness for our reliability-focused evaluation. BIRD's evidence- (or hint-) based text-to-SQL setting is incompatible with our benchmark, as our task requires incorporating both feasible and infeasible questions as model input. For further discussion, please refer to GC2 in the general comment section above.

Q2: More details about the annotation process (number of annotators, expertise level, annotation procedure, ensuring consistency between annotators, quality assurance)

As summarized in GC3 of the general comments, our annotation process involved three annotators (the authors), all proficient in SQL. We followed a systematic procedure to ensure consistency and quality:

• Feasible Questions: One annotator reviewed and re-annotated question templates and SQL structures. Two annotators corrected natural language paraphrases. All three annotators met to resolve disagreements and ensure consensus.
• Infeasible Questions: Annotators used specific keywords to modify feasible questions, creating infeasible ones. Real-time collaboration ensured consistency and high-quality annotations.

For more details, please refer to the revised manuscript.

Q3: The advantage of the proposed infeasible data creation instead of template-based method

Our keyword-based question creation method combines the strengths of template-based and template-free annotation methods. Annotators are provided with specific keywords and sample feasible questions. They modify these questions to make them infeasible by incorporating the keyword's intent. This approach introduces greater semantic diversity and creates infeasible questions that closely resemble real user questions, enhancing the dataset's realism and the model's ability to handle complex scenarios.

Template-based method:

Consider this infeasible question template: "When is the next earliest hospital visit of patient 0000?"—where no record exists for the next hospital appointment in the database. Possible paraphrases generated from this template can be the following:

• "What is the soonest upcoming hospital visit scheduled for patient 0000?"
• "When is the next scheduled appointment for patient 0000?" While this method can effectively handle common unanswerable questions (missing-schema), the diversity of the question pool generated using this method may be limited.

Keyword-based method (this work):

Suppose the keyword "appointment" is provided, along with sample feasible questions. The task is to modify these questions to include the keyword, making them infeasible. Sampled feasible questions:

• "Has patient 0000 gotten any medication this year?"
• "Provide the count of hospital visits for patient 0000." Annotated infeasible questions (the infeasible keyword is now inserted):
• "Has patient 0000 gotten any medication this year and do they have any upcoming appointments?"
• "Provide the count of hospital visits for patient 0000 including any scheduled upcoming appointments." By incorporating the keyword into existing feasible questions to make them infeasible, this method ensures both semantic diversity and guarantees that the annotated questions are indeed infeasible. This approach allows us to create a wider range of infeasible questions that closely resemble real user questions.

Q10: Rationale behind using TriageSQL for training infeasible question classifiers (INF+)

Our dataset does not include infeasible questions in the training set, and TriageSQL is the only openly available dataset for question classification in database question answering. Therefore, we used it to train the infeasible question classifier. In task-oriented dialog systems, out-of-intent questions (i.e., infeasible questions) are typically excluded from the training set [1,2]. Meanwhile, using external datasets is common in the era of pre-trained models. For those who prefer not to use TriageSQL to fine-tune their models, in-context learning is a viable alternative, as demonstrated in one of our GPT-4o-based baselines.

[1] Zheng, Yinhe, Guanyi Chen, and Minlie Huang. "Out-of-domain detection for natural language understanding in dialog systems." IEEE/ACM Transactions on Audio, Speech, and Language Processing 2020. [2] Marek, Petr, Vishal Ishwar Naik, Anuj Goyal, and Vincent Auvray. "OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation." NAACL 2021 Industry track.

Q11: The term sub-model

Sub-models refer to the individual models in the CL-based text-to-SQL pipeline. To clarify, we have updated the manuscript: "These models operate sequentially, with task-specific sub-models functioning in the following order: infeasible question detection, SQL generation, and SQL error detection."

Q12: Meaning of big \Phi

Φ is defined in Formula (1).

Thank you for the detailed review of our paper and for engaging in the discussion with us. Please find our response below.

Q1: Annotation quality concerns

Regarding concerns about annotation quality, TrustSQL consists of three components: feasible data, infeasible data, and corresponding databases. First, the databases we use are openly available and feature complex schemas compared to others in the text-to-SQL literature (see comparison in GC1), which are far from the bias you mentioned. Second, the feasible data include questions, SQL queries, and evidence. These questions and SQL queries are sourced from existing datasets—we are not their original creators but act as reviewers and correctors. This approach is more like a cross-project review/annotation process rather than a single-project effort, which helps mitigate concerns about annotation bias compared to other works. Lastly, for infeasible data, we introduce infeasible keywords (detailed in Appendix A.3.1–A.3.3) and incorporate them into feasible questions to make them infeasible. Below, we summarize our annotation process for infeasible questions (detailed in Appendix A.3) and explain how we ensure that adding infeasible keywords consistently results in infeasible questions.

Missing-schema

Infeasible questions belonging to the missing-schema category are annotated based on hypothetical columns (columns that do not exist in the actual databases) listed in Appendix A.3.1. These columns are designed to mislead the model, and they are uniquely written for each table. For example, we create an infeasible question using one of the infeasible keywords “NUM_LOUNGE” in the airport table from ATIS. Starting with a feasible question like “What are the flights that leave from DAL and go to other airports?,” the annotators modify it by incorporating the meaning of the keyword to form a natural question, such as “Find the average number of lounges across all airports.” If the annotated question correctly references these keywords, we can ensure that it is infeasible because the referenced columns do not exist in the actual database.

Ambiguous and Non-SQL

The keywords in these categories are not database-specific but are instead general keywords that make questions infeasible. For ambiguous questions, given a list of feasible questions, the annotators are tasked with inserting vague words (e.g., “suitable,” “best”) or referentially ambiguous terms (“those,” “this”) (details in Appendix A.3.2). For non-SQL questions, task-related keywords (e.g., “clustering,” “sentiment analysis”) are provided, and the annotators modify questions by incorporating these keywords naturally (details in Appendix A.3.3).

After all, the quality check involves verifying whether the keywords are truly infeasible and ensuring that the annotated questions properly reflect these keywords.

Regarding the full taxonomy, we initially included Table 2 to provide a brief overview of the categories of infeasible questions, with more detailed taxonomies for each type presented in Table 8 (ambiguous) and Table 9 (non-SQL). However, we plan to include a unified and comprehensive taxonomy of question types in the revised manuscript.

Q2: Contributions and comparison with existing works

As the title of the paper suggests, our work aims to measure “text-to-SQL reliability.” To evaluate reliability in more realistic scenarios, we consider complex databases (↑ schema size), questions requiring complex SQL queries (↑ #Tab/DB and ↑ Avg SQL Tok), and a new task setting where user questions include both feasible (AnserQ) and infeasible (UnansQ and AmbigQ) cases. Additionally, we introduce an evaluation penalty (Eval Penalty) to address varying user safety requirements. At the core of our work is answering this critical question: ”Is my model reliable enough for safe deployment compared to others?”—quantified by calculating the difference between helpfulness (i.e., the number of correct model decisions) and harmfulness (i.e., the number of incorrect decisions weighted by a penalty). No existing single work directly addresses this question in the context of safe text-to-SQL model deployment.

Q3: The choice of SQLCoder

Thanks to your suggestion, we have avoided using the term 'SOTA' when referring to SQLCoder in the revised manuscript. Instead, we describe it as a decoder-only, SQL-specialized Code Llama. Regarding your comment on SQLCoder's PostgreSQL setting, we fine-tune SQLCoder as a decoder-only model on the training portion of TrustSQL, so its original SQL dialect is not a major concern for us. Of course, using CodeS or other models could be another option for fine-tuning a decoder-only model, and it might perform better on SQL generation. However, we believe this change would not significantly affect the claims made in our paper. What contributes most to this benchmark is the use of abstention strategies, especially as the penalty increases.

Q4: Diversity of infeasible question and novelty of this work

Thanks to your suggestion, we have included a comparison between keyword-based and template-based (EHR-SQL) infeasible data generation in Appendix A.4. While extensive coverage of infeasible questions is valuable, we argue that it does not fundamentally alter our main claim. Our primary focus is to evaluate the reliability of text-to-SQL models in answering the question, “Is my text-to-SQL model the best and reliable enough for safe deployment compared to others?”—rather than simply classifying input question types, as done by TriageSQL and DTE. Infeasible questions are incorporated to simulate a more realistic evaluation setting, where not all questions are feasible, and we annotate these questions based on common types observed in previous user studies.

Since you question the diversity of infeasible questions compared to other works, we provide a more detailed discussion below:

TriageSQL’s infeasible question types:

• Improper: Random utterances.
• ExtKnow: Questions that cannot be answered using the given schema.
• Ambiguous: Ambiguity in column references.
• Non-SQL: Operations beyond SQL’s scope.

DTE’s infeasible question types:

• Column ambiguity: Ambiguity in column references.
• Column unanswerable: Questions that cannot be answered using the given schema.
• Additional types mentioned in DTE, such as value ambiguity, value unanswerable, calculation unanswerable (requiring external knowledge of formulas), and out-of-scope questions (beyond SQL’s scope), are excluded in DTE as they are found to be less frequent in their user study.

Infeasible question types in our benchmark compared to TriageSQL and DTE:

• Missing-schema: Corresponds to ExtKnow and Ambiguous in TriageSQL and to column unanswerable and ambiguity in DTE.
• Ambiguous: Covers referential ambiguity and vagueness*. While DTE and TriageSQL limit ambiguity to column-related issues, we include different types but adhere to the same principle: questions cannot be feasible without further clarification.
• Non-SQL: Matches the out-of-scope category in DTE and the non-SQL category in TriageSQL.

Categories not included in our benchmark and reasons for exclusion:

• Improper (TriageSQL): Our preliminary analysis showed that random utterances sampled from other datasets are too easily filtered out to add as a meaningful category.
• Value ambiguity/unanswerable (DTE): Determining whether a specific value exists in the database exceeds the scope of a single-turn text-to-SQL setting.
• Calculation unanswerable (DTE): With models like GPT-4o already having extensive knowledge of formulas, labeling such questions as unanswerable is no longer appropriate.

* One fundamental cause of infeasible questions is the ambiguity that arises from the richness of natural language expressions and the habitual omission of details by users [1, 2].

Q5: Choice of Metric (c)

As the title of our paper suggests, penalty-based scoring is a key contribution of our work. Existing text-to-SQL approaches do not penalize incorrect outputs, thereby overlooking the importance of model abstention in model development. By introducing this metric into text-to-SQL evaluation, we provide a way to measure user-defined reliability—specifically, whether the model's helpfulness outweighs its harmlessness—an aspect that no other text-to-SQL benchmark currently offers.

To answer the question "how the metric was developed," we adapt a metric originally proposed in Reliable VQA [3] for text-to-SQL tasks and extend it to address unanswerable questions. To answer "how the metric was validated," our experiments demonstrate how the reliability score (RS) correlates with coverage and risk, making it a practical tool for selecting models under different user-defined safety requirements. Regarding "its implications for benchmarking," we observed that models like GPT-4o perform well with low penalties, while T5-based models with threshold mechanisms excel in high-penalty settings due to their conservative prediction strategies—findings that would not have been discovered without this benchmark. We believe this contribution is essential for the adoption of text-to-SQL models in real-world industries with varying safety needs.

[1] Wang et al., "Know what I don’t know: Handling ambiguous and unknown questions for text-to-sql." ACL Findings 2023.
[2] Radhakrishnan et al., "ColloQL: Robust text-to-SQL over search queries." IntEx-SemPar 2020.
[3] Whitehead et al., “Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly.” ECCV 2022.

Q6: Domain bias

Our benchmark includes three significantly distinct database domains: airline travel, education, and healthcare. The goal of our work is to evaluate the reliability of models on complex databases and SQL queries, particularly in scenarios where not all questions are feasible. We believe that as long as methodological trends remain consistent across domains, this level of domain diversity is sufficient to validate our claims and conclusions. Indeed, we observed consistent modeling trends across databases in our experiments.

To clarify, our work does not include cross-domain text-to-SQL experiments where no in-domain training data is provided and the number of domains may be a primary concern. As noted in prior work, "the zero-shot setting is overly restrictive compared to how text-to-SQL systems are likely to be used in practice [4]." Similarly, we argue that achieving high performance in the current cross-domain text-to-SQL generalization setting is not the only challenge in this field. Instead, our goal is to develop or select the best models that meet a certain safety standard for text-to-SQL systems with some in-domain data available, which is a more realistic setting in practice.

Regarding domain biases in infeasible questions, we consider only the "missing-schema" type (Appendix A.3.1) to be database-specific, while ambiguous and non-SQL questions are not, as they are annotated with general, database-independent keywords (Appendix A.3.2-3). However, we believe this does not impact the models' behavior across databases. None of the models in our experiments are manually tailored to specific domains. We use the same model architecture and learning algorithm across all three databases, and the conclusions remain consistent.

If you still consider domain bias a concern in our work, we welcome your input on the number and variety of domains needed to sufficiently support our claims.

[4] Lee et al., "KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers." ACL 2021.

Thank you for taking the time to review our paper. Please find our responses below.

Q1: Comparison with existing work

Please refer to GC1 in the general comment section above for a detailed comparison table illustrating how TrustSQL differs from other benchmarks. In summary, TrustSQL uniquely integrates multiple types of input questions—including unanswerable, ambiguous, and answerable questions—and includes executable SQL with associated databases, features that many existing benchmarks lack.

Q2: Quality metric for dataset and annotation procedure

Since the dataset was not created through crowdsourcing, we did not report inter-annotator agreement statistics. Instead, the three authors of this paper, all proficient in SQL, served as annotators. We ensured high-quality annotations by resolving all disagreements until reaching consensus. The detailed annotation process is summarized in GC3 in the general comments section above.

Q3: Limited evaluation scope and generalizability

TrustSQL includes complex databases with an average of 18 tables per database and complex SQL queries involving multiple table joins (please refer to GC1 for more details). This complexity allows us to evaluate the reliability of text-to-SQL models more effectively. Regarding generalizability, the inclusion of the three domain-specific databases aims to evaluate whether methodological trends for reliable text-to-SQL modeling remain consistent across databases, rather than to assess a model's generalization ability in SQL generation across diverse databases. For further discussion, please refer to GC2 above.

Q4: Types of infeasible questions

As noted in our paper, it is not feasible to cover all possible types of infeasible questions in a benchmark dataset. Instead, we focus on the most problematic types identified in previous studies and conduct annotations based on these observations. We argue that if a model struggles with these types, it is unlikely to handle other question types effectively. While we considered adding more types of infeasible questions—including the categories you suggested—ambiguities in evaluation led us to select the current infeasible categories.

Regarding the three classes you mentioned, our responses are as follows:

• Inaccurate Premises: In text-to-SQL, feasibility is determined by whether the necessary information exists in the database schema, regardless of the factual accuracy of the premise. Therefore, the factual accuracy of a question is not critical in determining its feasibility.
• Malformed Queries: Determining whether a question with typos or grammatical errors remains well-formed introduces ambiguity. It becomes challenging to decide if such a question should be classified as ambiguous or unanswerable. To maintain clear evaluation boundaries, we chose to exclude such cases but acknowledge this as an area for future work.
• Requiring External Knowledge: To ensure a fair comparison across models with varying levels of world knowledge, we excluded questions requiring external knowledge. We focused on knowledge explicitly provided in the SQL assumptions for each database (Appendices A.1.1–A.1.3).

Q5: Recommended strategies for selecting c

We have elaborated on the selection of c in the general comments section above (see GC4). In summary, the choice of c depends on the safety requirements for model deployment and can vary based on user preferences, SQL proficiency, or organizational policies. We provide guidelines to assist users in selecting an appropriate c value for their specific needs.

Thank you for taking the time to review our paper. Please find our responses below.

Q1: Analysis of abstention mechanisms

Coverage and risk are valuable metrics for evaluating the correctness ratio of models; however, the Reliability Score (RS) offers a more comprehensive measure of reliability under a fixed penalty setting. Since our focus is on comparing model performance across varying penalty settings, coverage and risk alone do not provide sufficient insight for making informed model selection decisions.

An important takeaway is that when the penalty for incorrect decisions is low, seemingly high-performing models like GPT-4o-powered baselines perform well because they maintain low risk while offering reasonable coverage. However, in scenarios with extremely high penalties, models with adjustable thresholds, such as T5-3B leveraging uncertainty estimation of the internal model state, are more suitable. This is because threshold adjustments enable finer control over decisions to abstain or answer, allowing the modeler to align the model’s behavior with specific safety requirements. In single-turn text-to-SQL scenarios where mistakes are unacceptable, setting stricter thresholds ensures that only SQL generation outputs with high certainty are considered—an essential feature for high-stakes applications.

Q2: Guidelines for selecting the penalty parameter c

We have elaborated on the selection of the penalty value c in the general comments section (see GC4). In summary, the choice of c depends on the safety requirements for model deployment, which can vary based on user preferences, SQL proficiency, or organizational policies. To assist users, we provide guidelines to help them select an appropriate c value tailored to their specific needs.

Thank you for taking the time to review our paper. Please find our responses below.

Q1: Re-annotation details should be included in the main content rather than the appendix

We agree that the re-annotation process is an important part of our paper and appreciate your suggestion. While we initially adhered to the 9-page recommendation from the Call for Papers website (https://iclr.cc/Conferences/2025/CallForPapers), we have now expanded the main content to 10 pages. Specifically, we have added detailed descriptions of the data annotation process in Section 4. This includes comprehensive information about the annotation methodology, the roles of the annotators, and the steps taken to ensure data quality.

Q2: The rationale behind the choice of c

We have elaborated on the selection of the penalty value c in the general comments section (see GC4). In summary, the choice of c depends on the safety requirements for model deployment and can vary based on user preferences, SQL proficiency, or organizational policies. We provide guidelines to assist users in selecting an appropriate c value for their specific needs.

Q3: Annotator details and resolving annotation inconsistencies

Three authors of this paper served as annotators, performing manual annotation and review to ensure the highest possible data quality (no crowdsourcing was conducted). During the review process, all annotators met in person to resolve annotation disagreements until consensus was reached. We have included common areas of annotation disagreement in the revised manuscript. For a brief summary of this process, please refer to GC3 in the general comments section above.

Q4: How was clarity ensured during annotation, and what metric was used to determine overly similar SQL?

We ensured clarity by verifying whether questions accurately reflect their corresponding SQL queries, without introducing implicit assumptions beyond the SQL assumption text (Appendix A.1.1–A.1.3). Each question-SQL pair was manually reviewed by the annotators at the sample level. To identify overly similar SQL queries, we first categorized question templates that use the same placeholders (e.g., "Tell me flights from city_name1 to city_name0" and "What are airlines that provide services from city_name1 to city_name0," which share the placeholders city_name1 and city_name0). Templates with the same placeholders were then checked to determine whether they have identical SQL conditions and logical structures. If they were found to be semantically identical, we merged the templates.