Structure-Guided Large Language Models for Text-to-SQL Generation

Dear Reviewer EsJa,

Thanks a lot for your careful review and professional comments. We are truly grateful for your expertise and constructive feedback that help us advance our research.

Regarding W1: Performance on BIRD and SPIDER: "The primary limitation of this approach lies in the experimental section, where some of the most recent state-of-the-art methods, such as CHESS, Distillery, and CHASE-SQL, are not included. "

A: Thanks a lot for your careful review and professional comments.

1. Clarification of experiment setting

To avoid any potential confusion, we first clarify several important points:

• Our model, like most text-to-SQL methods, is model-agnostic, meaning that it can be integrated with any LLM as the backbone model.
• In our original submission, we reported results using GPT-3.5 and GPT-4 for cost-effectiveness considerations, with the BIRD results in Table 2 specifically using GPT-4.
• While current top performances on the BIRD leaderboard are achieved using more advanced LLMs such as Gemini-1.5 Pro and GPT-4o.

2. Performance comparison on BIRD

To ensure a fair comparison, we have conducted additional experiments using different LLMs as backbones.

1) Baseline collection

We added top-performing models from the BIRD leaderboard as baselines since the SPIDER leaderboard has remained static since February 2024. Specifically, we compared SGU-SQL against two categories of methods: (i) Open-source models with available paper and codes: MAC-SQL, Super-SQL, E-SQL and CHESS; and (ii) Undisclosed methods that have demonstrated strong performance: PURPLE, Distillery and CHASE-SQL.

Table 1: Performance comparison on BIRD dev with different LLMs as backbones.

Note that PURPLE, Distillery, and CHASE-SQL are closed-source models. We will update their results on GPT-4 and GPT-4o once their implementations become publicly available.

2) Performance analysis As shown in Table 1, we have the following observations:

• Using GPT-4 as the backbone, SGU-SQL achieves the best performance compared to other models using the same backbone.
• With GPT-4o, SGU-SQL achieves 69.28% in terms of execution accuracy, outperforming the strong baselines: PURPLE (68.12%), CHESS (68.31%), E-SQL (65.58%) and Distillery (67.21%). (We didn’t compare our model with CHASE-SQL+GPT-4o since CHASE-SQL is still closed-source and unable to integrate other LLMs.)
• The SOTA model (the only model outperforming ours) is CHASE-SQL+Gemini-1.5 Pro (73.14%). Google Research implemented this model using Gemini 1.5 Pro, and they just released the paper in October 2024, with the code still unavailable.

3. Performance comparison on SPIDER

1) Baseline collection In Table 1 of the original submission, we evaluated SGU-SQL on SPIDER with 11 text-to-SQL methods from 2 categories:

• Traditional structure-based methods: RAT-SQL, LGESQL, RESDSQL, Graphix-T5
• LLM-based models: DTS-SQL, CodeS, C³-SQL, Super-SQL, EPI-SQL, DIN-SQL, DAIL-SQL

To provide a more comprehensive comparison, we have added top-performing models from the BIRD leaderboard as new baselines, including PURPLE, MAC-SQL, CHESS and CHASE-SQL.

Table 2: Performance comparison on SPIDER in terms of Execution Accuracy.

2) Performance analysis

As shown in Table 2, SGU-SQL consistently achieves superior performance compared to the strongest baselines. This performance advantage is particularly significant given that it surpasses both traditional text-to-SQL architectures and recent LLM-enhanced approaches, validating the effectiveness of our method.

Manuscript Revision: We have added the results to Tables 1, 2 and 5 of our updated manuscript. Besides that we included the additional experiments with different LLMs as backbones in Section C.3 (line994-1017).

Regarding W2: Table 2&3 are not in the main content. "Crucial tables, such as Table 2 and Table 3, have been relegated to the appendix."

A: Thanks for your careful review. Following your suggestion, we have relocated the key results table to the main body of the revised manuscript to improve readability.

Regarding the performance of SGU-SQL+Gemini-1.5 Pro: ”Additionally, while it is true that the current SOTA approach on the BIRD leaderboard, CHASE-SQL, uses the Gemini-1.5-pro model and their code is not available for implementation with the GPT-4o model, I would appreciate it if you could report the performance of your approach using the Gemini-1.5-pro model instead. Since your work is model-agnostic, this would help avoid any confusion about the SOTA claims of your approach.”

A: Thank you for making this valuable suggestion. We are now working on the experimental results of this backbone LLM and will come back as soon as possible with a complete comparison. The new results and the discussions will be included in the final revision.

Regarding the performance on Spider: ”On the Spider benchmark, it is important to note that the SOTA model on the test set achieves an execution accuracy of 91.2, which is higher than all the baselines considered in the comparison.“

A: Thanks a lot for your expertise and careful review.

The SPIDER leaderboard has remained static since February 2024, with only a select number of recent papers reporting their performance on this benchmark. Our comparison encompasses a comprehensive range of methods, including:

• Top-performing models in Spider leaderboard：DAIL-SQL (2th, 3th), DIN-SQL (4th), C3-SQL (6th), RESDSQL(8th).
• Recent innovations: DTS-SQL, CodeS, Super-SQL, EPI-SQL, PURPLE, MAC-SQL, CHESS and CHASE-SQL.

Note that Miniseek (1st) is the top-performing model on the Spider leaderboard, which achieves an execution accuracy of 91.2. We didn't include this model since it is still anonymous, with both the paper and code unavailable.

As shown in Table 4, SGU-SQL consistently achieves superior performance compared to the strongest baselines on SPIDER in terms of Execution Accuracy.

Table 4: Performance comparison on SPIDER in terms of Execution Accuracy.

Regarding the performance of CHASE-SQL and CHESS on Spider: ”Furthermore, recent approaches like CHASE-SQL and CHESS did not perform model fine-tuning or prompt optimization for their performance reports on the Spider benchmark, as their goal was to demonstrate the generalization capabilities of their methods. This makes the comparison with your approach somewhat inequitable.”

A: Thank you for raising this important point about experimental fairness. We have carefully reviewed the methodologies described in both papers:

• In CHASE-SQL (Section 4.3): The authors mentioned that they assess the generalizability of the proposed CHASE-SQL by evaluating it in an end-to-end way without modifying the few-shot samples in the prompts or training a new selection model.
• CHESS (Section 4.2.3): The author mentioned that without specifically fine-tuning a new model for candidate generation or modifying the in-context learning samples.

After careful review, we believe the comparisons remain fair and valid for the following reasons:

• Fairness: Our SGU-SQL uses no candidate generation, model selection, or carefully generated few-shot examples. As such, all methods are evaluated under comparable conditions without task-specific optimizations.
• Effectiveness: The primary contribution of SGU-SQL is its structure-guided decomposition strategy. The results, without relying on few-shot examples or candidate selection, still effectively demonstrate the merits of our core contribution.

Thank you again for your insightful feedback, which has helped strengthen our manuscript considerably. We deeply value your expertise and thoroughness, and we are eager to address any remaining questions you may have.

Dear Reviewer EsJa,

Thanks a lot for providing such thorough and constructive feedback. Besides, we are grateful for your recognition of our efforts in improving this paper, and also greatly appreciate the opportunity to clarify the remaining concerns and further strengthen our work. Below are detailed responses to your comments and suggestions:

Regarding the performance on BIRD: ”The methods considered for comparison are those with published papers, but there are other approaches without published papers that have achieved higher performance than SGU-SQL using the GPT-4o model. For example, DSAIR + GPT-4o and OpenSearch-SQL, v2 + GPT-4o, achieved performance scores of 74.32 and 69.30, respectively.”

A: Thanks a lot for your expertise and careful review. To make a comprehensive comparison, we first have a careful review of the BIRD leaderboard.

Table 1: Top-performing text-to-SQL models ranked by BIRD leaderboard.

*Results reported in the paper.

As shown in Table 1, the current BIRD leaderboard features a mix of academic research and industrial solutions. For a comprehensive evaluation, we selected baselines from two categories:

• Open-source models with published papers and available code (MAC-SQL, Super-SQL, E-SQL, CHESS).
• High-performing methods with only available papers (PURPLE, Distillery, CHASE-SQL).

To summarize, from the top-8 methods in the BIRD leaderboard, we included PURPLE (8th), CHESS (6th), Distillery (5th), and CHASE-SQL (1st) in our comparisons. We excluded the remaining 4 methods (DSAIR, ExSL, OpenSearch-SQL, Insights AI) since they are all industrial solutions without any released instructions (papers and technical reports) or accessible code.

Table 2: Performance comparison on BIRD dev with different LLMs as backbones.

To make it clearer, we reorganized these results in the following table to compare their overall performance.

Table 3: Performance comparison on BIRD dev.

As shown in Tables 2 and 3, we have the following observations:

• The SOTA model is CHASE-SQL+Gemini-1.5 Pro \textcolor{maroon}{(73.14\\%)}. Google Research implemented this model using Gemini 1.5 Pro, and they released the paper in October 2024, with the code still unavailable.
• SGU-SQL+GPT-4o achieves \textcolor{maroon}{69.28\\%}, outperforming the strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%). (We didn’t compare our model with CHASE-SQL+GPT-4o since CHASE-SQL is still closed-source and unable to integrate other LLMs.)
• SGU-SQL+GPT-4 achieves the best performance compared to the other baselines using GPT-4 as the backbone.

Dear Reviewer EsJa,

Thank you for your expertise and insightful comments. During the past few days, we have conducted additional experiments to compare our model with CHASE-SQL by using more advanced LLMs as backbones.

Comparision with CHASE-SQL: ”Additionally, while it is true that the current SOTA approach on the BIRD leaderboard, CHASE-SQL, uses the Gemini-1.5-pro model and their code is not available for implementation with the GPT-4o model, I would appreciate it if you could report the performance of your approach using the Gemini-1.5-pro model instead. Since your work is model-agnostic, this would help avoid any confusion about the SOTA claims of your approach.”

A: Thank you for your expertise and insightful comments. Following your suggestion, we have conducted additional experiments to compare our model with CHASE-SQL by using more advanced LLMs as backbones.

Model Selection

• Gemini 1.5 Pro: When conducting experiments with Gemini 1.5 Pro, we noted that there are currently 5 available versions: gemini-1.5-pro-latest, gemini-1.5-pro, gemini-1.5-pro-001, gemini-1.5-pro-002 and gemini-1.5-pro-exp-0827. Since CHASE-SQL did not specify its model version, we conducted experiments with two versions - gemini-1.5-pro and gemini-1.5-pro-latest - for a comprehensive comparison.
• Claude 3.5 Sonnet: Additionally, as CHASE-SQL reported performance with Claude 3.5 Sonnet, we also included it as a backbone model.

Performance Analysis

Table 1: Performance on BIRD-dev in terms of Execution Accuracy.

Due to time and API budget limits, we have currently only evaluated our model's performance with Gemimi 1.5 Pro and Claude 3.5. We plan to conduct more comprehensive experiments with other baselines using these advanced LLMs in future work.

The experiment results demonstrate several key insights:

• First, when using Gemini 1.5 Pro as the backbone, SGU-SQL achieves highly competitive results (72.76% with gemini-1.5-pro and 72.93% with gemini-1.5-pro-exp-0827) compared to CHASE-SQL (73.01%).
• Furthermore, with Claude 3.5 Sonnet as the backbone, SGU-SQL (70.36%) slightly outperforms CHASE-SQL (69.53%). This improvement suggests that our method may better leverage Claude's capabilities through its structured decomposition approach.
• Our approach demonstrates robust and competitive performance across different state-of-the-art language models.

Thanks again for your expertise and professional comments. Your suggestions and guidance have been invaluable in shaping our revisions. Should there be any remaining concerns, we would be more than happy to provide additional information or explanations to assist in your review process.

Dear Reviewer EsJa,

We would like to express sincere gratitude for the time and effort you have invested in reviewing our paper. Your insights and suggestions have been invaluable in helping us improve the quality of our work. As the deadline for the Rebuttal approaches, we want to reach out and inquire whether our latest responses have addressed your concerns. If there are any remaining concerns or areas that require further clarification, we would greatly appreciate it if you could provide us with additional feedback.

Best regards,

Authors of Paper7028

Dear Reviewer EsJa,

Thanks again for taking the time to review our paper and providing valuable feedback. We have carefully responded to your latest concerns, conducting extensive experiments comparing our approach with CHASE-SQL on BIRD. We have also provided detailed analysis by using different advanced LLMs as backbones. We hope these analyses can address your concerns.

As the author-reviewer discussion deadline approaches in a few hours, we want to reach out and inquire whether our responses have addressed your concerns. If there are any remaining concerns or areas that require further clarification, we would greatly appreciate it if you could provide us with additional feedback.

Best regards,

Authors of Paper7028

Dear Reviewer vRvu,

Thank you for your recognition of our work and for providing such thorough and insightful feedback. Your comments and suggestions are invaluable in helping us improve the quality and clarity of our work.

Regarding W1: Additional experiment on SPIDER-2. "The framework is novel and intuitive but because of SPIDER not being the best representation of production grade SQL, it would also beneficial to probably test them using SPIDER-2. BIRD is somewhat closer to a real-world SQL but SPIDER-2 has large-scale and diverse databases which evaluates the proposal on a much better dataset revealing how the proposal works on large scale datasets. "

A: Thank you for making this valuable suggestion. SPIDER-2 is currently the most challenging text-to-SQL dataset available, which consists of there versions:

Table 1: Data statistics of SPIDER-2.

Spider 2.0 is designed to evaluate agent-based frameworks in enterprise-level text-to-SQL workflows. Since our SGU-SQL is a pure text-to-SQL model rather than an agent-based system, we conducted our evaluation on Spider 2.0-Snow and Spider 2.0-Lite. These two variants maintain the challenging aspects of SQL generation while focusing on pure text-to-SQL generation tasks without requiring complex agent-based planning and interactions.

Table 2: Execution accuracy for baseline methods on Spider 2.0-Snow and Spider 2.0-Lite.

As shown in Table 2, while the performances are relatively low across all models, SGU-SQL consistently demonstrates better capability in handling complex SQL generation tasks in both single and multi-database scenarios.

Manuscript Revision: Thanks for your suggestion. We have added these additional experiments and analyses in Section D.1 (line1021-1025) of our updated submission.

Regarding W2,Q1,Q2: Future work. "There's not much information provided on the future direction of this work which might hurt the significance/potential of the work a little. I'm mainly looking for how can this work be extended as it can provides opportunities for other researchers trying to explore in the same direction. In a real world, the databases are changed very often and there may be challenges where the proposed framework might not work same as proposed. In such cases, what components of the framework can still be relevant and what components must be changed is particularly industrial practitioners care about. How does this framework vary for dialects other than SQL? ”

A: Thank you for this thoughtful feedback regarding future directions. Discussing potential extensions is crucial for the research community. Following your suggestion, we have identified several promising future research directions from the following there perspectives:

1. Technical Extensions

a. Structure-aware few-shot example selection

Motivation: While our framework emphasizes the significance of the decomposition strategy, we recognize that the performance of LLM-based text-to-SQL can be further enhanced through tailored few-shot example selection. Current approaches to few-shot example selection primarily rely on keyword matching and semantic similarity between user queries. These surface-level matching approaches often fail to identify the most effective examples because they consider only query semantics while ignoring the underlying SQL structural complexity.

Solution: One promising solution is to incorporate syntax structure information into the few-shot example selection process. This structure-aware approach would consider both semantic relevance and SQL structural patterns, enabling better matching of complex query requirements with appropriate examples.

Research questions: How could we get reliable and distinguishable syntax structures? How to effectively measure the similarity between different syntax structures?

Dear Reviewer vRvu,

We would like to express sincere gratitude for the time and effort you have invested in reviewing our paper. Your insights and suggestions have been invaluable in helping us improve the quality of our work. If there are any remaining concerns or areas that require further clarification, we would greatly appreciate it if you could provide us with additional feedback.

Best,

Authors of Paper7028

Dear Reviewer b8Gr,

We are deeply grateful for your recognition of our work and also appreciate your time and effort in providing insightful suggestions that can help further polish our paper. Below are detailed responses to your comments and suggestions:

Regarding Q1: Comparison with CoT and Few-shot based methods: "Have the authors compared the current method with the traditional CoT or Few-shot methods?"

A: Thank you for raising this important point. To verify the effectiveness of SGU-SQL, we compared it to the state-of-the-art baselines with different prompting strategies, including (i) CoT-based models: DIN-SQL and DTS-SQL, and (ii) Few-shot example selection methods: DAIL-SQL, CodeS and Super-SQL, in Table 1 and Table 2 of the original submission.

To strengthen our evaluation, we have added new comparisons with more recent baselines, including (i) CoT-based models: ACT-SQL and MAC-SQL, and (ii) Few-shot example selection methods: MCS-SQL and PURPLE.

Table 1: Performance comparison between SGU-SQL and new baseline methods on SPIDER.

As shown in Table 1, our SGU-SQL achieves 87.95% execution accuracy on SPIDER-dev, surpassing both traditional CoT and Few-shot approaches. This superior performance demonstrates that our structure-guided approach provides advantages over conventional prompting strategies.

Manuscript Revision: We have added the results in Table 5 to the updated manuscript, and the analysis can be found in Section C.2 (line972-992) of our revision.

Regarding Q2: Implemention details of LLM fine-tuning. "What Lora rank does the authors use for fine-tuning? Have they tested it on multiple ranks?"

A: Thank you for the insightful comment. In our experiments, we used 8 as the default rank since it provided the best balance between performance and computational efficiency. Specifically, we experimented with multiple LoRA ranks (4, 8, 16, 32) and found that higher ranks (16, 32) showed diminishing returns while significantly increasing memory usage, while Lower ranks (4) resulted in slightly degraded performance.

Regarding Q3: Implemention details of structure linking: "Could the authors provide more details about the GNN used in structure linking?"

A: Thank you for the insightful comment. Following your suggestion, we add more analysis for graph-based structure linking.

Training details

• During each training iteration, we first sample a batch of query nodes, where each query node $a$ is paired with one positive match $s$ from the database schema and multiple negative matches $\{sₖ\}$. For each pair, we construct an enclosing subgraph G(a,s) that encompasses the query graph, neighboring nodes of $s$, and their connecting edges.
• Then, we employ RGAT as the backbone model to process these subgraphs. RGAT learns the structural information within each subgraph to generate meaningful representations ${h_i}^k$ that capture the compatibility between natural language concepts and database elements. The matching score is then measured by the degree of compatibility (defined in Eq.16).
• Finally, we optimize the model using a contrastive loss function (Eq.17). This training objective encourages the model to maximize scores for correct matches while minimizing scores for incorrect ones. Through this process, the matching scores evolve into reliable indicators that effectively guide the selection of appropriate database elements during SQL generation.

Ablation study of RGAT

To further verify the effectiveness of the backbone model, i.e., RGAT, we replace it with other alternatives, including RGCN [1] and CompGCN [2].

Table 3: Alation study and structure linking models.

As shown in Table 3, our RGAT-based approach outperforms alternative architectures across all evaluations. Besides that, removing structure-aware linking causes a dramatic performance drop - accuracy decreases by 5.33% on SPIDER-dev and 6.49% on BIRD-dev. These substantial reductions highlight the critical role of our structure-aware linking strategy.

[1] Modeling Relational Data with Graph Convolutional Networks.

[2] Composition-based Multi-Relational Graph Convolutional Networks.

Manuscript Revision: We added more details about the design principles and training process of the structure linking in Section 3.1.3 (line222-278) of our updated submission.

Dear Reviewer b8Gr,

We would like to express sincere gratitude for the time and effort you have invested in reviewing our paper. Your insights and suggestions have been invaluable in helping us improve the quality of our work.

Best,

Authors of Paper7028

Regarding W2-a: Performance on BIRD. “Table 2 is BIRD performance (exe acc 61), which is lower than SOTA 74 https://bird-bench.github.io/. SGU-SQL focus on leveraging structure-aware links between queries and database schema, so to prove effectiveness of the method, the method need to be tested on more challenging database, like BIRD.”

A: Thank you for the constructive comments. To avoid any potential confusion, we first offer the following clarification:

• Our model, like most text-to-SQL methods, is model-agnostic, meaning that it can be integrated with any LLM as the backbone model.
• In our original submission, we chose to report results using GPT-3.5 and GPT-4 for cost-effectiveness considerations, with the BIRD results in Table 2 specifically using GPT-4.
• While current top performances on the BIRD leaderboard are achieved using more advanced LLMs such as Gemini-1.5 Pro and GPT-4o.

To ensure a fair comparison, we have conducted additional experiments using different LLMs as backbones.

Baseline collection

For a thorough evaluation of SGU-SQL's performance, we added top-performing models from the BIRD leaderboard as baselines since the SPIDER leaderboard has remained static since February 2024. Specifically, we compared SGU-SQL against two categories of methods: (i) Open-source models with available paper and codes: MAC-SQL, Super-SQL, E-SQL and CHESS; and (ii) Undisclosed methods that have demonstrated strong performance: PURPLE, Distillery and CHASE-SQL.

Table 1: Performance comparison on BIRD dev with different LLMs as backbones.

Note that PURPLE, Distillery, and CHASE-SQL are closed-source models. We will update their results on GPT-4 and GPT-4o once their implementations become publicly available.

Performance analysis As shown in Table 1, our SGU-SQL achieves competitive performance across different LLM backbones. Specifically, we have the following observations:

• Using GPT-4 as the backbone, SGU-SQL achieves the best performance compared to other models using the same backbone.
• With GPT-4o, SGU-SQL achieves 69.28% in terms of execution accuracy, outperforming several strong baselines: PURPLE (68.12%), CHESS (68.31%), E-SQL (65.58%) and Distillery (67.21%).
• The only model showing higher performance is CHASE-SQL (released in October 2024), which uses Gemini 1.5 Pro as its backbone. Notably, CHASE-SQL incorporates a query fixer module that leverages database execution feedback to guide LLMs to iteratively refine generated queries. In contrast, our model generates SQL queries in a single pass without utilizing any execution feedback.

Manuscript Revision: We added the additional experiments with different LLMs as backbones in Section C.3 (line994-1017).

Regarding W2-b: Performance on SPIDER: "Table 1 Spider. Many of the baseline methods are not Text-to-SQL specific methods, therefore performance is not competitive. additionally only show method is good on Spider is not enough. ideally test performance on test split by submitting to leaderboard for generation ability."

A: We really appreciate your suggestion to further compare our model with more text-to-SQL models. In Table 1 of the original submission, we evaluated SGU-SQL on SPIDER with 11 text-to-SQL methods from 2 categories:

• Traditional structure-based methods: RAT-SQL, LGESQL, RESDSQL, Graphix-T5
• LLM-based models: DTS-SQL, CodeS, C³-SQL, Super-SQL, EPI-SQL, DIN-SQL, DAIL-SQL

To provide a more comprehensive comparison, we have added top-performing models from the BIRD leaderboard as new baselines, including PURPLE, MAC-SQL, CHESS and CHASE-SQL. As shown in Table 2, SGU-SQL consistently achieves superior performance compared to the strongest baselines on SPIDER in terms of Execution Accuracy. While the Spider leaderboard is no longer actively maintained, we'll further validate our model's performance by submitting results to the BIRD leaderboard.

Table 2: Performance comparison between our proposed SGU-SQL and the strongest baselines on SPIDER in terms of Execution Accuracy.

Manuscript Revision: We added the results to Tables 1, 2 and 5 of our updated manuscript.

Dear Reviewer 53mq,

Thanks a lot for your detailed feedback. We really appreciate your time and effort in pointing out the potential concerns related to our paper, and also, thanks a lot for the opportunity to further clarify the technical details and contribution of our framework.

Regarding W1: Methodology and Preliminary Section is not well-organzied. “It is not very clear how schema-linkage is used in SQL generation by decomposition (sec 3.1). How is score eq21 used in your algorithm? Preliminary Section has detailed equations, etc, however this section seems not help introduce the method, may confuse readers.”

A: Thank you for making these valuable comments. We first provide a brief recap of the key idea of our framework.

The Key Idea of Our Framework

SGU-SQL offers significant improvements over existing approaches in two key aspects.

• Syntax-based prompting. While traditional methods attempt to generate entire SQL queries in one step or rely on simple decomposition strategies, SGU-SQL breaks down the complex generation task in a syntax-aware manner. This ensures that the generated queries maintain both semantic accuracy (correctly capturing user intentions) and syntactic correctness (following proper SQL structure).
• Combine with structure-aware linking. Our framework's unique combination of structure-aware linking and syntax-guided decomposition creates a robust bridge between user intentions and database structures. The structure-aware linking ensures that each natural language concept is correctly mapped to the corresponding database elements, while the syntax-guided decomposition ensures these mappings are properly utilized in constructing the SQL query.

The Effect of Structure Linking

Now, we provide the following clarification about how structure-aware linking is used in decomposition-based SQL generation:

• Initial Decomposition Phase: The structure-aware linking mechanism creates a foundation for accurate decomposition by establishing precise mappings between natural language elements and database schema components. By understanding which parts of the user query correspond to specific database tables, columns, and relationships, the system can make informed decisions about how to break down the SQL generation task.
• Recursive Generation Phase: The linking results serve as contextual guides for generating each SQL component in the decomposed subtasks. For example, given a subquery "Find the names of students who take CS courses", the structure-aware linking could provide the relevant contextual information like "students" maps to the Student table, "CS courses" maps to the Course table with a condition of ”department = 'computer science'”.

The Clarification of Matching Score (Eq. 21)

Definition of Matching Score

The matching score in Eq.21 (Eq.16 in our updated manuscript) serves as a fundamental component in our algorithm by measuring the likelihood of mappings between natural language concepts and database elements. When a user mentions "student name" in the question, the matching score helps determine whether this should map to the "name" column in the "Student" table versus other potential tables and columns in the database.

Training details

The training of matching score consists of three key steps:

• First, during each training iteration, we sample a batch of query nodes, where each query node 'a' is paired with one positive match 's' from the database schema and multiple negative matches {sₖ}. For each pair, we construct an enclosing subgraph G(a,s) that encompasses the query graph, neighboring nodes, and their connecting edges.
• Second, we employ RGAT as the backbone model to process these subgraphs. RGAT learns the structural information within each subgraph to generate meaningful representations hᵢᵏ that capture the compatibility between natural language concepts and database elements. The matching score is then measured by the degree of compatibility using Eq.21 (Eq.16 in our updated manuscript).
• Finally, we optimize the model using a contrastive loss function (Eq.16 in our updated manuscript). This training objective encourages the model to maximize scores for correct matches while minimizing scores for incorrect ones. Through this process, the matching scores evolve into reliable indicators that effectively guide the selection of appropriate database elements during SQL generation.

Manuscript Revision: We added more details about the design principles and training process of the structure linking and matching score in Section 3.1.3 (line258-280) of the updated manuscript. Besides that, we replaced the technical-heavy Preliminary Section with a more focused Problem Statement Section (line95-107), moving the detailed equations and technical preliminaries to Section A (line741-800) of our updated manuscript.

Dear Reviewer 53mq,

Thanks a lot for your timely response and careful review. We are delighted that you found the new ablation studies are helpful. We also greatly appreciate the opportunity to further clarify the remaining concerns.

Clarification

To avoid any potential confusion, we first offer the following clarification:

• The SOTA model (the only model outperforming ours) is CHASE-SQL+Gemini-1.5 Pro \textcolor{maroon}{(73.14\\%)}. Google Research implemented this model using Gemini 1.5 Pro as its backbone, and they just released this paper in October 2024, with the code still unavailable.
• SGU-SQL+GPT-4o achieves \textcolor{maroon}{69.28\\%}, outperforming the other strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%).

We provide detailed explanations and analysis as follows:

Regarding the performance on BIRD: ”However, my primary concern remains that the method's performance on BIRD is significantly lower (61 vs. SOTA at 74), a gap of 13%.“

Regarding your concern about the performance on BIRD, we have provided some related experiments and analysis in Table 1 of the previous response.

Table 1: Performance comparison on BIRD dev with different LLMs as backbones.

To make it clearer, we reorganized these results in the following table to compare their overall performance.

Table 2: Performance comparison on BIRD dev.

As shown in Tables 1 and 2, we have the following observations:

• The SOTA model (the only model outperforming ours) is CHASE-SQL+Gemini-1.5 Pro \textcolor{maroon}{(73.14\\%)}. Google Research implemented this model using Gemini 1.5 Pro, and they just released the paper in October 2024, with the code still unavailable.
• SGU-SQL+GPT-4o achieves \textcolor{maroon}{69.28\\%}, outperforming the strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%). (We didn’t compare our model with CHASE-SQL+GPT-4o since CHASE-SQL is still closed-source and unable to integrate other LLMs.)
• SGU-SQL+GPT-4 achieves the best performance compared to the other baselines using GPT-4 as the backbone.

Regarding the concern of possible overfitting on Spider: “In contrast, the model achieves SOTA on SPIDER. This disparity suggests the method may be heavily overfitted to SPIDER.”

As shown in Table 3, all models show a similar performance pattern across both datasets. This consistent pattern suggests that the variations in performance stem from BIRD's inherent complexity rather than model overfitting. (BIRD is currently the largest text-to-SQL dataset. It comprises over 12,751 question-SQL pairs, spanning 95 large databases with a total size of 33.4 GB.)

Table 3: Performance comparison in terms of Execution Accuracy.

Regarding the performance gap of using different LLMs: “I also find it unlikely that the use of a weaker model (GPT-4 in this method vs. GPT-4o in others) would result in a performance drop of 13% on BIRD. The significant drop in performance indicates some fundamental components may be missing in the method.”

Table 4: Performance gap of using different LLMs.

Looking at the results across all models, switching from GPT-4 to GPT-4o typically brings a 5-8% improvement:

• All models show similar improvements when switching to GPT-4o.
• Our model's improvement (+7.48%) aligns with this pattern.

Thanks again for your active response and professional comments. Your expertise and guidance have been invaluable in shaping our revisions. Should there be any remaining concerns, we would be more than happy to provide additional information or explanations to assist in your review process.

Dear Reviewer 53mq,

Thanks again for taking the time to review our paper and providing valuable feedback. We have carefully responded to your latest concerns, conducting extensive experiments comparing our approach with CHASE-SQL on BIRD. We have also provided detailed analysis by using different advanced LLMs as backbones. We hope these analyses can address your concerns.

As the author-reviewer discussion deadline approaches, we would be very grateful if you could share your valuable feedback. We are committed to strengthening our work and would be grateful for your further insights.

Best regards,

Authors of Paper7028

Dear Reviewer 53mq,

Thank you for your expertise and insightful comments. To achieve a thorough evaluation of SGU-SQL, we have conducted additional experiments to compare our model with CHASE-SQL by using more advanced LLMs as backbones during the past few days,

Model Selection

• Gemini 1.5 Pro: When conducting experiments with Gemini 1.5 Pro, we noted that there are currently 5 available versions: gemini-1.5-pro-latest, gemini-1.5-pro, gemini-1.5-pro-001, gemini-1.5-pro-002 and gemini-1.5-pro-exp-0827. Since CHASE-SQL did not specify its model version, we conducted experiments with two versions - gemini-1.5-pro and gemini-1.5-pro-latest - for a comprehensive comparison.
• Claude 3.5 Sonnet: Additionally, as CHASE-SQL reported performance with Claude 3.5 Sonnet, we also included it as a backbone model.

Performance Analysis

Table 1: Performance on BIRD-dev in terms of Execution Accuracy.

Due to time and API budget limits, we have currently only evaluated our model's performance with Gemimi 1.5 Pro and Claude 3.5. We plan to conduct more comprehensive experiments with other baselines using these advanced LLMs in future work.

The experiment results demonstrate several key insights:

• First, when using Gemini 1.5 Pro as the backbone, SGU-SQL achieves highly competitive results (72.76% with gemini-1.5-pro and 72.93% with gemini-1.5-pro-exp-0827) compared to CHASE-SQL (73.01%).
• Furthermore, with Claude 3.5 Sonnet as the backbone, SGU-SQL (70.36%) slightly outperforms CHASE-SQL (69.53%). This improvement suggests that our method may better leverage Claude's capabilities through its structured decomposition approach.
• Our approach demonstrates robust and competitive performance across different state-of-the-art language models.

Thanks again for your expertise and professional comments. Your suggestions and guidance have been invaluable in shaping our revisions. As the deadline for the Rebuttal approaches, we want to reach out and inquire whether our responses have addressed your concerns. If there are any remaining concerns or areas that require further clarification, we would greatly appreciate it if you could provide us with additional feedback.

Best,

Authors of Paper7028

Dear Reviewer 53mq,

We would like to deeply express our gratitude for your recognition of our efforts to address your concerns. Your expertise and constructive feedback really help us advance our research. We will add the additional experiments and discussion in our final revision.

Best regards,

Authors of Paper7028

Table 1: Top-performing text-to-SQL models ranked by BIRD leaderboard.

*Results reported in the paper.

To summarize, from the top-8 methods in the BIRD leaderboard, we included PURPLE (8th), CHESS (6th), Distillery (5th), and CHASE-SQL (1st) in our comparisons. We excluded the remaining 4 methods (DSAIR, ExSL, OpenSearch-SQL, Insights AI) since they are all industrial solutions without any released instructions (papers and technical reports) or accessible code.

b. Performance analysis

Table 2: Performance comparison on BIRD dev with different LLMs as backbones.

Due to time and API budget limits, we have currently only evaluated our model's performance with Gemimi 1.5 Pro and Claude 3.5. We plan to conduct more comprehensive experiments with other baselines using these advanced LLMs in future work.

As shown in Table 2, we have the following observations:

• SGU-SQL+GPT-4 achieves the best performance compared to the other baselines using GPT-4 as the backbone.
• SGU-SQL+GPT-4o achieves 69.28%, outperforming the strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%). (We didn’t compare our model with CHASE-SQL+GPT-4o since CHASE-SQL is still closed-source and unable to integrate other LLMs.)
• When using Gemini 1.5 Pro as the backbone, SGU-SQL achieves highly competitive results (72.76% with gemini-1.5-pro and 72.93% with gemini-1.5-pro-exp-0827) compared to CHASE-SQL (73.01%).
• With Claude 3.5 Sonnet as the backbone, SGU-SQL (70.36%) slightly outperforms CHASE-SQL (69.53%). This improvement suggests that our method may better leverage Claude's capabilities through its structured decomposition approach.

To summarize, our approach demonstrates robust and competitive performance across different state-of-the-art language models, which verifies the effectiveness of our proposed syntax-based prompting strategy.

Performance on Spider

a. Baseline selection

The SPIDER leaderboard has remained static since February 2024, with only a select number of recent papers reporting their performance on this benchmark. Our comparison encompasses a comprehensive range of methods, including:

• Top-performing models in Spider leaderboard：DAIL-SQL (2th, 3th), DIN-SQL (4th), C3-SQL (6th), RESDSQL(8th).
• Recent innovations: DTS-SQL, CodeS, Super-SQL, EPI-SQL, PURPLE, MAC-SQL, CHESS and CHASE-SQL.

Note that Miniseek (1st) is the top-performing model on the Spider leaderboard, which achieves an execution accuracy of 91.2%. We didn't include this model since it is still anonymous, with both the paper and code unavailable.

b. Performance analysis

Table 3: Performance comparison on SPIDER in terms of Execution Accuracy.

As shown in Table 3, SGU-SQL consistently achieves superior performance compared to the strongest baselines on SPIDER in terms of Execution Accuracy.

Thank you again for your time and expertise throughout this review process. We deeply appreciate your expertise and recognition of our work. Each point raised has given us new perspectives to consider. Hope that this summary could facilitate the next stage of review and discussion.

Best regards,

Authors of Paper7028