LAIA-SQL: Enhancing Natural Language to SQL Generation in Multi-Table QA via Task Decomposition and Keyword Extraction

Q5 Why is the proprietary model utilized in the module ablation study different? In Table 3, the comparison between "Entity Retrieval + Revision + Generation" and "Entity Retrieval + Generation" uses different models (GPT-4o and GPT-4), which is confusing. Did this setting lead to a correct conclusion in Section 5.3? Furthermore, how does the framework work with open-source models in each module?

A5 We have updated Table 3 to include results using GPT-4 as the base model. Here are the latest results:

We have double-checked and updated some numbers, but these changes do not affect the conclusions in Section 5.3. Additionally, Table 4 presents the performance of different open-source models in each module. Our findings show that using GPT-4o-mini (finetuned) for user question understanding, MinHASH and Stella-400M for entity retrieval, and GPT-4o for revision and generation yields the best results.

Q6 Will the authors consider submitting the results of the paper to the public leaderboards BIRD and Spider to verify their performance on the test set? I believe making the LAIA-NLU dataset publicly available would enhance the contribution of the paper.

A6 Of course. We have already contacted the team of bird leaderboard and submitted our code, we will release the leaderboard result publicly after the final decision made because of the anonymous policy.

Q3 Some of the contributions in the paper are not clearly verified. In L18-L20, the authors claim that current approaches in NL2SQL suffer from slow retrieval speeds, and in L27-L31 they claim to have achieved a speed-up. However, Table 2 presents only the total time cost without detailing the time cost of each component, particularly the entity retrieval module. The authors should present the retrieval performance of current approaches as a baseline and then show that their proposed framework is more effective in clearly verifying their contribution.

A3 Thank you for your insightful comment. We apologize for the confusion. Our intention was to express the total speed, not just the retrieval speed. Since not all methods utilize a retrieval function, it’s challenging to make direct comparisons specifically for the retrieval module. We have clarified this point in the revised paper by specifying "total speed." Thank you for bringing this to our attention.

Q4 How does the Revision module work? The authors claim that the incorrect SQL and execution feedback are combined to prompt LLMs to revise the original SQL. How many times is the revision conducted? Is there any criterion or threshold to prevent an infinite loop of revisions for extremely difficult questions? Additionally, how does this module relate to cost-effectiveness? Will the overall framework cost be influenced by the number of revisions?

A4 As we mentioned in our paper, the initial predicted SQL statements may include errors such as incorrect table names, misaligned columns, or extraneous symbols. Therefore, in the revision module, we feed the erroneous SQL statements along with their corresponding error messages back into the LLM for revision. This iterative process results in syntactically correct and operational SQL queries, ultimately yielding the correct answers.

Here is the prompt that we use in the revision module，which is also shown in the appendix:

'''

Objective: Your objective is to make sure a query follows the database admin instructions and use the correct conditions.

Database Schema

{DATABASE_SCHEMA}

Constraints

1. When you need to find the highest or lowest values based on a certain condition, using ORDER BY + LIMIT 1 is prefered over using MAX/MIN within sub queries.

2. If predicted query includes an ORDER BY clause to sort the results, you should only include the column(s) used for sorting in the SELECT clause if the question specifically ask for them. Otherwise, omit these columns from the SELECT.

3. Predicted query should return all of the information asked in the question without any missing or extra information.

4. For key phrases mentioned in the question, we have provided the most similar values within the columns denoted by "-- examples" in front of the corresponding column names. This is a crucial hint indicating the correct columns to use for your SQL query.

5. If you are joining multiple tables, make sure to use alias names for the tables and use the alias names to reference the columns in the query. Use T1, T2, T3, ... as alias names.

Question:

{QUESTION}

ERROR INFORMATION

{Error Infomation}

Steps that you should follow

{Main Task} {Sub Task} {Hint}

Predicted query:

{SQL}

Pay attention to the ERROR INFORMATION, based on the error revise the SQL query. Think about whether the predicted query used the hint and evidence already, if not, use the hint and evidence in the sql query generation.

Please respond with a JSON object structured as follows (if the sql query is correct, return the query as it is):

{{"revised_SQL": "Your revised SQL query is here."}}

'''

LAIA-SQL involves running the Revision module only once. While the module does improve accuracy, more revisions do not necessarily yield better results. We tested our method on 50 samples and varied the threshold for revisions (1 to 5). Here are the results:

We observed that, as the threshold increases, accuracy improves along with the cost, but the time does not follow a clear pattern. Setting the threshold at 3 provided the best balance of cost, accuracy, and time for our implementation.

In our method, revisions are controlled by a pre-set threshold to 1 to ensure there are no infinite loops, addressing both the accuracy and cost-effectiveness concerns.

Q2 The paper does not include an error analysis of the SQL generation task. Referring to advanced studies [6][7], an error analysis that categorizes the types of incorrect execution would provide significant insights into which types of SQL the proposed methods handle well and where they still struggle. Additionally, a comparison of execution errors with previous methods could clearly demonstrate the improvements brought by the proposed framework. Readers are often interested in understanding why it works as much as how it works.

A2 We appreciate the reviewer's suggestion on conducting an error analysis. Unfortunately, previous researches like CHESS and CHASE-SQL have not disclosed their specific datasets used for error analysis, making an objective comparison challenging. Therefore, we performed our own error analysis by randomly sampling 100 fail cases from the BIRD dataset without comparison.

As you can see in the following table, we observed that 45% of the golden SQL queries had issues, primarily with incorrect column names (11%) and missing GROUP BY/DISTINCT/RANK clauses (8%). It was also noted that 6% didn't follow the evidence cues. For LAIA-SQL, 49% of the generated SQL queries were incorrect, mainly due to not adhering to evidence (17%), incorrect column usage (11%), and incorrect operations (8%). Vague questions accounted for 6% of issues, where the question information was insufficient to generate correct SQL, affecting both golden and predicted SQL.

Our future work aims to enhance the framework's capability to understand and apply evidence during SQL generation. Detailed data is provided in the following table:

Here is the detailed error in predicted sql of LAIA-SQL:

ERROR 1 Incorrect Column Name

Question: "Please list the phone numbers of the direct charter-funded schools that are opened after 2000/1/1."

Evidence: "Charter schools refers to Charter School (Y/N) = 1 in the frpm"

GOLD_SQL: "SELECT T2.Phone FROM frpm AS T1 INNER JOIN schools AS T2 ON T1.CDSCode = T2.CDSCode WHERE T1.Charter Funding Type = 'Directly funded' AND T1.Charter School (Y/N) = 1 AND T2.OpenDate > '2000-01-01'"

PREDICTED_SQL: "SELECT T1.Phone FROM schools T1 JOIN frpm T2 ON T1.CDSCode = T2.CDSCode WHERE T1.FundingType = 'Directly funded' AND T1.OpenDate > '2000-01-01' AND T2.Charter School (Y/N) = 1;"

ERROR 2 Incorrect Aggregation: LAIA-SQL didn’t join the table correctly

Question: "Which school in Contra Costa has the highest number of test takers?"

Evidence: ""

GOLD_SQL: "SELECT sname FROM satscores WHERE cname = 'Contra Costa' AND sname IS NOT NULL ORDER BY NumTstTakr DESC LIMIT 1"

PREDICTED_SQL: "SELECT T1.sname FROM satscores T1 JOIN schools T2 ON T1.cds = T2.CDSCode WHERE T2.County = 'Contra Costa' AND T1.NumTstTakr IS NOT NULL ORDER BY T1.NumTstTakr DESC LIMIT 1;"

ERROR 3 Incorrect Evidence: LAIA-SQL did not follow the evidence to generate PREDICTED_SQL

Question: "Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type of these schools?"

Evidence: "Average of average math = sum(average math scores) / count(schools)."

GOLD_SQL: "SELECT T1.sname, T2.Charter Funding Type FROM satscores AS T1 INNER JOIN frpm AS T2 ON T1.cds = T2.CDSCode WHERE T2.District Name LIKE 'Riverside%' GROUP BY T1.sname, T2.Charter Funding Type HAVING CAST(SUM(T1.AvgScrMath) AS REAL) / COUNT(T1.cds) > 400"

PREDICTED_SQL: "SELECT T1.School, T1.FundingType FROM schools T1 JOIN satscores T2 ON T1.CDSCode = T2.cds WHERE T1.County = 'Riverside' GROUP BY T1.CDSCode HAVING AVG(T2.AvgScrMath) > 400;"

We thank the reviewer for their time and interest in our work. The detailed instructions provided helpful feedback to improve our work. We have addressed the reviewer’s questions and suggestions, and we provide a list of responses below:

Q1 The paper would benefit from a more comprehensive review of existing work in the field. Section 2 should include an individual subsection that covers more related work on NL2SQL, such as [1][2][3][4][5].

A1 I have incorporated additional related work into the revised version of the paper. Here is the updated Section 2:

"Table Question Answering (Table QA) is a task to help users who are not proficient in coding skill or advanced spreadsheet software retrieve complex table data by question answering Javaid et al. (2023); Al Naqbi et al. (2024). A leading approach in Table QA is Natural Language to SQL (NL2SQL), which translates natural language queries into SQL, allowing users to interact with databases in everyday language Gao et al. (2023); Hong et al. (2024); Liu et al. (2024)."

"Beyond directly applying large language models (LLMs) for NL2SQL, hybrid methods that combine LLMs with various modules have also shown promise. Notable examples include ADVETAPi et al. (2022), CHESS Talaei et al. (2024), PURPLE Ren et al. (2024), Din-SQL Pourreza & Rafiei (2024a), DTS-SQL Pourreza & Rafiei (2024b) and MAC-SQL Wang et al. (2023). Nevertheless, as demonstrated in Figure 1, challenges such as slow data retrieval, erroneous SQL code generation, and high operational costs still remain."

Thank you for your constructive feedback. We appreciate the opportunity to address your concerns.

1. Error Analysis & Comparison to CHESS:

We acknowledge your point regarding the sample size for error analysis. We adhered to the convention observed in similar works, such as CHESS [1], which utilized a sample of 147 instances. CHASE-SQL [2], on the other hand, did not specify their sample size for error analysis. Therefore, due to the limited time of rebuttal, we plan to utilized 100 instances, which take 20% of the total failed cases, for error analysis. If you still have concern on the instance number, we would like to seek your guidance on what you deem as a sufficiently rigorous sample size.

Regarding the accuracy comparison, it's noteworthy that the CHESS model refreshed an accuracy of 68.31% after our paper was submitted. In its paper, the accuracy is 65%. This achievement was largely due to its fine-tuned generation model. Its paper indicates that using the GPT-4-turbo generation model, it achieved an accuracy of 55.78%. In contrast, our model obtained an accuracy of 60.36% with GPT-4-turbo, surpassing CHESS. Moreover, our research emphasizes that focusing solely on accuracy is not sufficient. Our methodology excels not only in accuracy but also in reducing operational costs by up to 97% and runtime by 52.4%, making it highly suitable for large-scale NL2SQL applications in real-world scenarios.

Regarding the significant issue you mentioned, we would like to highlight that Table 4 in our paper clearly compares the performance of different open-source models, such as DeepSeek-Coder-V2-Instruct and DeepSeek-Coder-V2-Base, on SQL generation. Additionally, we have included a comparison of different embedding models, including Stella-400M and Stella-1.5B. Furthermore, Table 5 demonstrates the natural language understanding capabilities of these various open-source models like Llama3-8B, Baichuan2-7B, Mistral-7B and Baichuan2-13B. We hope these tables address your concerns and provide clarity on the strengths and weaknesses of the models we evaluated. We would also like to emphasize that comparing different open-source models is resource-intensive. Therefore, we have focused our comparisons on the top-performing models listed on the leaderboards to ensure a comprehensive yet efficient evaluation.

2. Typos and Grammar:

We appreciate your observation regarding typos. We have since corrected these errors. While we understand the importance of rigorous writing in top-tier conferences, we believe that the primary focus should be on the contributions of the work. That said, we will ensure thorough proofreading before the camera-ready submission to meet the highest standards of presentation.

3. Dataset Contribution:

We understand your concerns about the motivation and complexity of our framework. It is important to emphasize that our main contribution lies in the Dataset Track rather than direct competition with methods like CHESS. Our dataset, LAIA-NLU, has undergone three rounds of meticulous human verification, ensuring its high quality. This required significant human and material resources. This dataset addresses the existing gap in the field of table QA by providing validation data specifically for fine-grained multi-level task decomposition and keyword extraction.

Experimental results demonstrate that LAIA-NLU can effectively evaluate different models' understanding of user questions. Additionally, models fine-tuned on our dataset exhibit substantial improvements in SQL generation accuracy, resulting an improvement of accuracy from 59.62% to 67.28%. We believe LAIA-NLU has broad applicability, potentially benefiting future NL2SQL algorithm development and other applications involving user question understanding.

Thank you again for your valuable feedback. We believe that a positive and collaborative discussion can greatly contribute to the advancement of our field. We look forward to engaging in a meaningful dialogue to further improve our research! Have a nice day.

[1] Shayan Talaei, et al. "CHESS: Contextual Harnessing for Efficient SQL Synthesis" arXiv preprint, 2024.

[2] Mohammadreza Pourreza, et al. "CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL" arXiv preprint, 2024.

Dear Reviewer 3hsY,

Thank you once again for your valuable and constructive feedback. Your insightful comments have significantly contributed to enhancing the clarity and overall quality of our paper.

We believe that we have addressed all of your questions thoroughly. We are confident that the contributions of our work merit acceptance rather than rejection in dataset track. As the author-reviewer discussion period will end soon on November 26th, we would greatly appreciate it if you could take the time to review our comments and provide us with feedback. If there are any additional clarifications or experiments you would like us to conduct, please do not hesitate to let us know, as we are eager to demonstrate the merits of our paper.

Furthermore, we have incorporated the references you suggested into the manuscript, checked the author list and learned a lot from their works. Given your fairness and thoroughness as a reviewer, if our response addresses your concerns, we kindly ask you to consider raising the rating of our work.

Thank you once again for your dedication and support.

Best regards,

Authors of Paper 10678

Dear Reviewer 3hsY:

Thank you for your valuable feedback. We have carefully addressed all your concerns and deeply appreciate your thoughtful suggestion to revise the manuscript. As a submission in the dataset track, our paper introduces a dataset that has undergone multiple rounds of manual verification to ensure high quality, effectively addressing the limitations of existing datasets of a similar nature. Additionally, fine-tuning models with our dataset significantly enhances their natural language understanding capabilities. This improvement, in turn, boosts the accuracy of SQL query generation by the NL2SQL method, as well as increases efficiency and reduces associated costs. In response to the valuable feedback received, we have also incorporated additional experiments as suggested, thereby making our study more comprehensive and robust. We hope these revisions comprehensively address your comments and are happy to discuss any additional concerns you may have.

Best,

Authors of Paper 10678

We thank the reviewer for their time and interest in our work. The detailed instructions provided helpful feedback to improve our work. We have addressed the reviewer’s questions and suggestions, and we provide a list of responses below:

Q1.1.1 BIRD Benchmark Reporting:

A1.1.1 We have submitted our code to the BIRD dataset team and are currently awaiting their results. We understand the importance of reporting the official test scores and will update our paper accordingly once we receive the results. This may take some time, and we appreciate your patience.

Q1.1.2 Ablation Study in Table 3:

A1.1.2 We have revised Table 3 to ensure consistency by using GPT-4 as the backbone model across all configurations. The updated results are as follows:

We hope this clarifies the rationale behind our choices and addresses the confusion.

Q1.1.3 Configuration "Revision + Generation":

A1.1.3 We understand that entity retrieval is not always utilized in NL2SQL approaches, and we appreciate you pointing this out. Moving forward, we will place greater emphasis on user question understanding, SQL generation, and revision in our research. Your feedback is invaluable in helping us improve our work.

Q1.1.4 Statements on Performance:

A1.1.4 We have revised our statements to accurately reflect the performance of our method in the paper. The updated phrasing is: "Overall, our framework performs better compared to currently available open-source or published methods."

Q1.2 For 3, I'm mainly referring to Table 2 in the paper. The main issue is that the evaluation process of getting these metrics is not included in the paper. Excluding the context makes the numbers hard to interpret. What does accuracy = 0.8 mean? Likewise for Time (s) and Cost.

A1.2 We have revised our paper accordingly to provide clearer context for the metrics presented in Table 2. Specifically, we have updated the term 'accuracy' to 'dev EX' to maintain consistency with Table 1. In addition, we have included a detailed explanation in Section 5.1.

Here's a clarification of the metrics:

• dev EX represents the accuracy of generating the final SQL for 10 questions.
• Time (s) indicates the total time required to generate the final SQL for these 10 questions.
• Cost corresponds to the monetary cost incurred to generate the final SQL for the 10 questions.

Q4 What is the "Accuracy" metric in Table 2? How does it differ from dev EX in Table 1?

A4 We have updated the "Accuracy" metric in Table 2 to "dev EX" for consistency with Table 1.

Q5 In Figure 1, a comparison is shown between GPT-4o and the "ours" method for task decomposition and keyword extraction. However, from what I understand, "ours" also uses GPT-4o. What is the main difference?

A5 In Figure 1, GPT-4o refers to the base model. "Ours" represents the LAIA-NLUer model, which has been fine-tuned using the LAIA-NLU dataset for task decomposition and keyword extraction. We have revised the figure to clarify this distinction.

Thank you again for your suggestions. We understand your concerns about the motivation and complexity of our framework. It is important to emphasize that our main contribution lies in the Dataset Track rather than direct competition with methods like CHESS. Our dataset, LAIA-NLU, has undergone three rounds of meticulous human verification, ensuring its high quality. This required significant human and material resources. This dataset addresses the existing gap in the field of table QA by providing validation data specifically for fine-grained multi-level task decomposition and keyword extraction.

Experimental results demonstrate that LAIA-NLU can effectively evaluate different models' understanding of user questions. Additionally, models fine-tuned on our dataset exhibit substantial improvements in SQL generation accuracy, resulting an improvement of accuracy from 59.62% to 67.28%. We believe LAIA-NLU has broad applicability, potentially benefiting future NL2SQL algorithm development and other applications involving user question understanding.

Thank you again for your valuable feedback. We believe that a positive and collaborative discussion can greatly contribute to the advancement of our field. We look forward to engaging in a meaningful dialogue to further improve our research! Have a nice day.

Q2. The comparison of experimental results with prior work is potentially misleading.

A2 As mentioned in Table 2's caption, we used GPT-4o as the base model to test methods including CHESS and TA-SQL. Specifically, we replaced CHESS’s default GPT-4-Turbo with GPT-4o for a fair comparison. Under this setting, CHESS incurred a cost of 11 USD for 10 questions, whereas our method required only 0.32 USD. This demonstrates that, with the same base model and dataset, our approach outperforms others in terms of time, accuracy, and cost.

The results in Table 1 are sourced from BIRD and the Spider leaderboard. Testing all methods using GPT-4o would indeed be cost-prohibitive. Therefore, in Table 2, we tested different methods on the same 10 questions using the same base model (GPT-4o), showing that our method achieved the best accuracy.

Additionally, some methods mentioned in Table 1, such as Dail-SQL, were not included in Table 2 due to unsuccessful reproduction. For instance, the reproduction of Dail-SQL took 756 seconds for 10 questions with an accuracy of just 0.3, potentially leading to ambiguous results.

We thank the reviewer for their time and interest in our work. The detailed instructions provided helpful feedback to improve our work. We have addressed the reviewer’s questions and suggestions, and we provide a list of responses below:

Q1 The connection between this paper and prior work should be improved.

A1 We have add the previous work in related work, here is the content in the revised manusript:

"In the domain of table QA, researchers have also developed several methods for NLU, such as QDecomp Tai et al. (2023) and QPL Eyal et al. (2023), which investigate prompting LLMs for query decomposition. Meanwhile, methods like DARA Fang et al. (2024), Iterated Decomposition Reppert et al. (2023) show potentials for NLU. Additionally, datasets have been constructed, such as BREAK Wolfson et al. (2020). However, existing NLU datasets in the table QA domain tend to decompose tasks into very fine-grained components, thereby overlooking the primary requirements of the task. This approach lacks a comprehensive evaluation of the model’s capabilities in task decomposition. Moreover, applications of table QA require assessment of the model’s ability to extract relevant keywords from user queries related to database filter values, column names, or table names, a capability often missing from previous datasets."

A1.1Clarification on Query Decomposition and Keyword Extraction:

Our original statement was "Effective SQL generation requires the model to excel in natural language understanding (NLU), which can be divided into two areas: 1) fine-grained task decomposition and 2) precise keyword extraction." We did not intend to imply that keyword extraction and task decomposition are the primary methods for solving multi-table QA. Our main point is that improving SQL generation accuracy necessitates the model's strong performance in natural language understanding.

A1.2 Regarding Section 2.2:

We appreciate your feedback. Our intention was to provide a comprehensive overview of the developments in both general NLU and TableQA fields. We recognize that a good related work section should summarize the progression in the broader domain as well as in the specific sub-field. We aimed to highlight the advancements and existing challenges in NLU both generally and within the context of table QA.

A1.3 Quantitative Evaluation Metrics for NLU:

We acknowledge that our initial phrasing may not have been precise. Upon careful review, we have revised our statement to better reflect our analysis of existing research: "Although the BREAK dataset \cite{wolfson2020break} includes QA pairs that decompose questions in a highly granular manner, it lacks data pertinent to the primary task. Additionally, while it extracts keywords, these are neither comprehensive nor noise-free, making them unsuitable for evaluating the NLU performance in table QA."

A1.4 Reference Correction for TA-SQL:

We have corrected the reference to TA-SQL to [6] as suggested.

A1.5 Clarification on GraphRAG:

Our intention was to indicate that some researchers have applied the GraphRAG method to TableQA with notable success. To avoid any confusion, we have decided to remove this part from our text.

Dear Reviewer 7reZ:

Thanks again for all of your constructive suggestions, which have helped us improve the quality and clarity of the paper!

Since the author-reviewer discussion period will end soon on Nov 26th, we appreciate it if you take the time to read our comments and give us some feedback. Please don't hesitate to let us know if there are any additional clarifications or experiments that we can offer, as we would love to convince you of the merits of the paper. We appreciate your suggestions. If our response resolves your concerns, we kindly ask you to consider raising the rating of our work.

Thanks for your time and efforts!

Best,

Authors of Paper 10678

Dear Reviewer 7reZ:

Thank you for your valuable feedback. We have carefully addressed all your concerns and deeply appreciate your thoughtful suggestion to revise the manuscript. As a submission in the dataset track, our paper introduces a dataset that has undergone multiple rounds of manual verification to ensure high quality, effectively addressing the limitations of existing datasets of a similar nature. Additionally, fine-tuning models with our dataset significantly enhances their natural language understanding capabilities. This improvement, in turn, boosts the accuracy of SQL query generation by the NL2SQL method, as well as increases efficiency and reduces associated costs. In response to the valuable feedback received, we have also incorporated additional experiments as suggested, thereby making our study more comprehensive and robust. We hope these revisions comprehensively address your comments and are happy to discuss any additional concerns you may have.

Best,

Authors of Paper 10678

We thank the reviewer for their time and interest in our work. The detailed instructions provided helpful feedback to improve our work. We have addressed the reviewer’s questions and suggestions, and we provide a list of responses below:

After carefully analyzing all the revisions, we observed that there were significant improvements in the accuracy of examples. Specifically:

• 17.74% of the successful cases resolved the "no such column" issue, which involved correcting references to tables and columns in the SQL code.

• 4.84% of the successful cases addressed the "Near 's': Syntax Error" issue by changing the filtering value to the correct format.

• 1.61% of the successful cases fixed the "No Such Function" problem, where there was an incorrect function call in the SQL (e.g., using the strftime function instead of the year function, which does not exist).

• 75.81% of the successful cases resulted in correct SQL sentences after being generated a second time.

Here are specific examples illustrating these improvements:

### Example One

SQL Generation

• Result:"no such column: E1.link_to_event"

• Question: "Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event?"

• Evidence: "percentage = DIVIDE(SUM(cost where event_name = 'Yearly Kickoff'), SUM(cost)) * 100"

• GOLD_SQL: "SELECT CAST(SUM(CASE WHEN T1.event_name = 'Yearly Kickoff' THEN T3.cost ELSE 0 END) AS REAL) * 100 / SUM(T3.cost) FROM event AS T1 INNER JOIN budget AS T2 ON T1.event_id = T2.link_to_event INNER JOIN expense AS T3 ON T2.budget_id = T3.link_to_budget"

• PREDICTED_SQL: "SELECT (SUM(E1.cost) * 100.0 / (SELECT SUM(E2.cost) FROM expense E2)) AS percentage FROM expense E1 JOIN event EV ON E1.link_to_event = EV.event_id WHERE EV.event_name = 'Yearly Kickoff';"

Revision

• Result:"correct answer"

• Revised_SQL: "SELECT (SUM(E1.cost) * 100.0 / (SELECT SUM(E2.cost) FROM expense E2)) AS percentage FROM expense E1 JOIN budget B ON E1.link_to_budget = B.budget_id JOIN event EV ON B.link_to_event = EV.event_id WHERE EV.event_name = 'Yearly Kickoff';"

### Example Two

SQL Generation

• Result:"near\"s\": syntax error"

• Question: "Among the students from the Student_Club who attended the event"Women's Soccer", how many of them want a T-shirt that's in medium size?"

• Evidence: "Women's Soccer is an event name; T-shirt that is in medium size refers to t_shirt_size = 'Medium'"

• GOLD_SQL: "SELECT COUNT(T1.event_id) FROM event AS T1 INNER JOIN attendance AS T2 ON T1.event_id = T2.link_to_event INNER JOIN member AS T3 ON T2.link_to_member = T3.member_id WHERE T1.event_name = 'Women''s Soccer' AND T3.t_shirt_size = 'Medium'"

• PREDICTED_SQL: "SELECT COUNT(*) AS medium_tshirt_count FROM attendance T1 JOIN event T2 ON T1.link_to_event = T2.event_id JOIN member T3 ON T1.link_to_member = T3.member_id WHERE T2.event_name = 'Women's Soccer' AND T3.t_shirt_size = 'Medium';"

Revision

• Result:"correct answer"

• Revised_SQL: "SELECT COUNT(*) AS medium_tshirt_count FROM attendance T1 JOIN event T2 ON T1.link_to_event = T2.event_id JOIN member T3 ON T1.link_to_member = T3.member_id WHERE T2.event_name = 'Women''s Soccer' AND T3.t_shirt_size = 'Medium';"

### Example Three

SQL Generation

• Result:"no such function: year"

• Question: "List all patients who first came to the hospital after year 1992 with prothrombin time (PT) level that are normal."

• Evidence: "first came to the hospital after year 1992 refers to year(First Date) > 1992; prothrombin time (PT) level that are normal refers to PT < 14;"

• GOLD_SQL: "SELECT T1.ID FROM Patient AS T1 INNER JOIN Laboratory AS T2 ON T1.ID = T2.ID WHERE STRFTIME('%Y', T1.First Date) > '1992' AND T2.PT < 14"

• PREDICTED_SQL: "SELECT T1.ID FROM Patient T1 JOIN Laboratory T2 ON T1.ID = T2.ID WHERE year(T1.First Date) > 1992 AND T2.PT < 14;"

Revision

• Result:"correct answer"

• Revised_SQL: "SELECT T1.ID FROM Patient T1 JOIN Laboratory T2 ON T1.ID = T2.ID WHERE strftime('%Y', T1.First Date) > '1992' AND T2.PT < 14;"

### Example Four

SQL Generation

• Result: "incorrect answer"

• Question: List circuits which host 4 f1 races from year 1990 to 2000."

• Evidence: "first came to the hospital after year 1992 refers to year(First Date) > 1992; prothrombin time (PT) level that are normal refers to PT < 14;"

• GOLD_SQL: "SELECT T1.ID FROM Patient AS T1 INNER JOIN Laboratory AS T2 ON T1.ID = T2.ID WHERE STRFTIME('%Y', T1.First Date) > '1992' AND T2.PT < 14"

• PREDICTED_SQL: "SELECT T1.ID FROM Patient T1 JOIN Laboratory T2 ON T1.ID = T2.ID WHERE year(T1.First Date) > 1992 AND T2.PT < 14;"

Revision

• Result: "correct answer"

• Revised_SQL: "SELECT T1.ID FROM Patient T1 JOIN Laboratory T2 ON T1.ID = T2.ID WHERE strftime('%Y', T1.First Date) > '1992' AND T2.PT < 14;"