Structure-Guided Large Language Models for Text-to-SQL Generation

Dear Reviewer jn4B, Thank you for your recognition of our work and for providing such thorough and insightful feedback. Your comments and suggestions are invaluable in helping us improve the quality and clarity of our work.

---

1. Insufficient details for error analysis. Thank you for your thorough review. In this paper, we perform an error analysis to evaluate our model’s performance. Specifically, we classify errors into two primary categories:

• Schema-linking errors: Incorrect matching of tables or columns in the database schema.
• Syntactic errors: Invalid SQL syntax, including misuse or omission of key SQL clauses (e.g., JOIN, GROUP BY, nested queries and others).

As illustrated in Figure 2 of our submission, we provided a detailed analysis of the error distribution. Compared to the baseline model, our approach significantly improves schema-linking accuracy and syntactic correctness.

---

2. The definition of VES metric is not clear. The definition of VES metric is not clear. Sorry for the confusion caused. Valid Efficiency Score (VES) is defined to measure the efficiency of valid SQL queries, which was first defined in BIRD bechmark.

A valid SQL query is a predicted SQL whose executed results exactly match the ground truth results. Specifically, VES evaluates both the efficiency and accuracy of predicted SQL queries. For a text dataset with $N$ examples, VES is computed by: $\text{VES} = \frac{1}{N}\sum_{n=1}^{N}**I**(V_n, \hat{V}_n) \cdot **R**(Y_n, \hat{Y}_n),$ where $\hat{Y}_n$ and $\hat{V}_n$ are the predicted SQL query and its executed results and $Y_n$ and $V_n$ are the ground truth SQL query and its corresponding executed results, respectively. $\text{I}(V_n, \hat{V}_n)$ is an indicator function, where: $**I**(V_n, \hat{V}_n) = 1 if V_n = \hat{V}_n.$ Then, $**R**(Y_n, \hat{Y}_n) = \sqrt{E(Y_n)/E(\hat{Y}_n)}$ denotes the relative execution efficiency of the predicted SQL query in comparison to ground-truth query, where $E(\cdot)$ is the execution time of each SQL in the database.

---

3. The examples of the syntax tree are not clear. Thanks a lot for your careful review. Following your suggestion, we have updated Figure 5 to include more complex examples, particularly those involving nested queries. We will add the revised figure in our new manuscript since we are not allowed to include external links in this version.

---

4. Personalization gaps. We sincerely appreciate your insightful suggestion regarding personalized decomposition strategies tailored to different user intents and database structures. This is indeed a promising research direction that could significantly enhance real-world applicability. Moving forward, we plan to extend our framework by incorporating adaptive decomposition mechanisms to further improve system performance.

---

5. Efficiency trade-offs. Thank you for making this valuable suggestion. To assess our approach thoroughly, we conducted the efficiency analysis on the BIRD dataset (33.4 GB total). Given that the queries in this dataset are categorized into 3 difficulty levels: simple, moderate, and challenging, we specifically tested our model on the challenging set of the BIRD dataset and compared its performance with DIN-SQL and MAC-SQL.

Table 1: Efficiency analysis on the ''Challenging'' set of BIRD.

As shown in Table 4, our model demonstrates superior performance while maintaining competitive computational efficiency. This superior efficiency can be attributed to our graph-based architecture. While baseline methods avoid the overhead of graph construction, they heavily rely on prompt-based modules that require multiple calls to LLMs like GPT-4. These API calls introduce substantial latency that accumulates during both the training and inference phases. In contrast, our graph-based approach, despite its initial graph construction overhead, achieves faster end-to-end processing by minimizing dependence on time-consuming API calls.

---

6. The performance of lightweight LLMs. Thanks for the valuable comments. Following your suggestion, we add the QwenCoder series model as the backbone LLM.

Table 2: Performance on BIRD with Qwen2.5-Coder as the backbone LLM.

XiYanSQL-QwenCoder series model is the SOTA method that uses lightweight Qwen2.5-Coder as the backbone. As shown in Table 2, our SGU-SQL outperforms this competitor across all model sizes, suggesting the effectiveness and robustness of our framework.

Dear Reviewer wUD6,

We are deeply grateful for your recognition of our work and also appreciate your time and effort in providing insightful suggestions that can help further polish our paper. Below are detailed responses to your comments and suggestions:

---

1. The effect of the base LLMs. Our model, like most text-to-SQL methods, is model-agnostic, meaning that it can be integrated with any LLM as the backbone model. To verify the effect of the base LLMs, we added additional experiments using GPT-4, GPT-4o, and Gemini-1.5 Pro as backbones.

Table 1: Performance comparison on BIRD dev with different LLMs as backbones.

Note that PURPLE, Distillery, and CHASE-SQL are closed-source models. We will update their results on GPT-4 and GPT-4o once their implementations become publicly available.

As shown in Table 1, our SGU-SQL achieves competitive performance across different LLM backbones. Specifically, we have the following observations:

• Using GPT-4 as the backbone, SGU-SQL achieves the best performance compared to other models using the same backbone.
• With GPT-4o, SGU-SQL achieves 69.28% in terms of execution accuracy, outperforming several strong baselines: PURPLE (68.12%), CHESS (68.31%), E-SQL (65.58%) and Distillery (67.21%).
• The only model showing higher performance is CHASE-SQL, which uses Gemini 1.5 Pro as its backbone. Notably, CHASE-SQL incorporates a query fixer module that leverages database execution feedback to guide LLMs to iteratively refine generated queries. In contrast, our model generates SQL queries in a single pass without utilizing any execution feedback.

---

2. Justification on the backbone GNN model. Thanks for your insightful comments. To verify the effectiveness of the backbone model, i.e., RGAT, we replace it with other alternatives, including RGCN [1] and CompGCN [2].

Table 2: Alation study on backbone GNN models.

As shown in Table 3, our RGAT-based approach outperforms alternative architectures across all evaluations. Besides that, removing structure-aware linking causes a dramatic performance drop - accuracy decreases by 5.33% on SPIDER-dev and 6.49% on BIRD-dev. These substantial reductions highlight the critical role of our structure-aware linking strategy.

[1] Modeling Relational Data with Graph Convolutional Networks.

[2] Composition-based Multi-Relational Graph Convolutional Networks.

---

3. Paper structure. Thanks a lot for your valuable suggestion. We will reorganize the paper and move the key experiments into the main content of the paper to enhance clarity.

Dear Reviewer 5QmZ,

Thank you for your expertise and insightful comments. Below are detailed responses to your comments and suggestions:

---

1. Performance on BIRD. Thanks for your insightful comments. For a thorough evaluation of SGU-SQL's performance, we added top-performing models from the BIRD leaderboard as baselines. From the top 10 methods in the BIRD leaderboard, we include CHASE-SQL (4th), OpenSearch-SQL (6th), Distillery (7th), CHESS (8th), and PURPLE (10th) in our comparisons. We exclude the remaining methods (AskData, Contextual-SQL, ExSL, Insights AI) since they are all industrial solutions without any released instructions (papers and technical reports) or accessible code.

Table 1: Performance comparison on BIRD dev with different LLMs as backbones.

Due to time and API budget limits, we have currently only evaluated our model's performance with Gemimi 1.5 Pro and Claude 3.5. We plan to conduct more comprehensive experiments with other baselines using these advanced LLMs in future work.

As shown in Table 1, we have the following observations:

• SGU-SQL+GPT-4 achieves the best performance compared to the other baselines using GPT-4 as the backbone.
• SGU-SQL+GPT-4o achieves \textcolor{maroon}{69.28\\%}, outperforming the strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%).
• When using Gemini 1.5 Pro as the backbone, SGU-SQL achieves highly competitive results (\textcolor{maroon}{72.76\\%}, with gemini-1.5-pro and \textcolor{maroon}{72.93\\%} with gemini-1.5-pro-exp-0827) compared to CHASE-SQL (73.01%).
• With Claude 3.5 Sonnet as the backbone, SGU-SQL (\textcolor{maroon}{70.36\\%}) slightly outperforms CHASE-SQL (69.53%).

To summarize, our approach demonstrates robust and competitive performance across different base LLMs.

---

2. The importance of schema linking. We thank the reviewer for raising this important point. While we commend Distillery-SQL’s novel schema-free paradigm leveraging iterative refinement and execution feedback, we respectfully contend that explicit schema linking remains indispensable for real-world text-to-SQL systems, particularly for three reasons:

• While Distillery-SQL achieves strong results without dedicated schema linking, its reliance on iterative query refinement (via augmentation/selection/correction) introduces substantial computational overhead. For instance, their pipeline requires multiple LLM calls with database execution feedback, which incurs significant latency and infrastructure costs. In contrast, schema linking modules enable single-pass query generation while maintaining desirable performance.
• Inaccurate schema linking degrades LLM-based SQL generation while accurate schema linking still improves the model performance. Current top-performing models (XiYAN-SQL, CAHSE-SQL, etc.) universally incorporate schema linking, achieving superior performance on benchmarks like BIRD and Spider (+5-8% over schema-free baselines). This aligns with our findings: explicit schema linking improves robustness, particularly for long-tail schemas and compositional queries.

---

3. Compared to CHASE-SQL. Thanks for your insightful comments. Following your suggestion, we compare our model with CHASE-SQL by integrating different backbone LLMs.

Notably, CHASE-SQL incorporates a query fixer module that leverages database execution feedback to guide LLMs to refine generated queries iteratively. In contrast, our model generates SQL queries in a single pass without utilizing any execution feedback. As shown in the table, our model shows more desirable performance than CHASE-SQL. It is because that traditional methods attempt to generate entire SQL queries in one step or rely on simple decomposition strategies, SGU-SQL breaks down the complex generation task in a syntax-aware manner. This ensures that the generated queries maintain both semantic accuracy (correctly capturing user intentions) and syntactic correctness (following proper SQL structure).

---

4. The structure of Table 1. Thanks a lot for your insightful comments. Following your suggestion, we will make Table 1 more concise by removing some less important baselines.

Dear Reviewer hLFK,

Thanks a lot for your detailed feedback. We really appreciate your time and effort in pointing out the potential concerns related to our paper, and also, thanks a lot for the opportunity to clarify the technical details and contribution of our framework.

To avoid any potential confusion, we first offer the following clarification:

• Our model, like most text-to-SQL methods, is model-agnostic, meaning that it can be integrated with any LLM as the backbone model.
• In Tables 1 and 2, we report the main results using GPT-3.5 and GPT-4 for cost-effectiveness considerations.
• We compare with the top-performing methods (CHASE-SQL, CHESS, Distillery) using Gemini-1.5 Pro and GPT-4o as the backbone LLM in Table 5 of the Appendix.

Below are our responses in detail.

---

1. Detailed comparison with XiYan-SQL. Thanks for your insightful comments. XiYan-SQL is the top research-based method (3rd on the BIRD leaderboard, behind two industrial solutions), but its source code and backbone LLM remain undisclosed.

However, the authors released the pre-train version (XiYanSQL-QwenCoder) on Hugging Face, enabling a direct comparison by using Qwen2.5-Coder as the backbone LLM.

Table 2: Comparison with XiYan-SQL using Qwen2.5-Coder as the backbone LLM.

As shown in Table 2, our SGU-SQL outperforms this competitor across all model sizes, suggesting the effectiveness and robustness of our framework.

2. Compared with other SOTA methods on BIRD. Thanks for your insightful suggestions. For a thorough evaluation of SGU-SQL's performance, we added top-performing models from the BIRD leaderboard as baselines. From the top 10 methods in the BIRD leaderboard, we include XiYan-SQL (3rd), CHASE-SQL (4th), OpenSearch-SQL (6th), Distillery (7th), CHESS (8th), and PURPLE (10th) in our comparisons. We exclude the remaining 4 methods (AskData, Contextual-SQL, ExSL, Insights AI) since they are all industrial solutions without any released instructions (papers and technical reports) or accessible code.

Table 1: Performance comparison on BIRD dev with different LLMs as backbones.

Note that PURPLE, Distillery, and CHASE-SQL are closed-source models. We will update their results on GPT-4 and GPT-4o once their implementations become publicly available.

As shown in Table 1, we have the following observations:

• SGU-SQL+GPT-4 achieves the best performance compared to the other baselines using GPT-4 as the backbone.
• SGU-SQL+GPT-4o achieves \textcolor{maroon}{69.28\\%}, outperforming the strong baselines: E-SQL+GPT-4o (65.58%), Distillery+GPT-4o (67.21%), PURPLE+GPT-4o (68.12%) and CHESS+GPT-4o (68.31%). (We didn’t compare our model with CHASE-SQL+GPT-4o since CHASE-SQL is still closed-source and unable to integrate other LLMs. While XiYan-SQL is also closed-source, only their Qwencoder series model has been released.)
• When using Gemini 1.5 Pro as the backbone, SGU-SQL achieves highly competitive results (\textcolor{maroon}{72.76\\%}, with gemini-1.5-pro and \textcolor{maroon}{72.93\\%} with gemini-1.5-pro-exp-0827) compared to CHASE-SQL (73.01%).
• With Claude 3.5 Sonnet as the backbone, SGU-SQL (\textcolor{maroon}{70.36\\%}) slightly outperforms CHASE-SQL (69.53%). This improvement suggests that our method may better leverage Claude's capabilities through its structured decomposition approach.

To summarize, our approach demonstrates robust and competitive performance across different base LLMs.

---

---

3. The effect of schema linking. Thank you for the constructive comments. Following your suggestion, we compare our graph-based schema linking with previous models and report the results in the following table.

Table 3. Schema linking on BIRD.

As shown in Table 3, our model achieves the best performance across different linking strategies, which further verifies the effectiveness of our proposed structure-aware linking mechanism.

---

4. Typos in Definition 1. Thanks for your careful review. We will fix the typos in line62-63 by changing the statement “Given a natural language query D and a database schema Q” into “Given a natural language query Q and a database schema D.”