READ-SQL: Reasoning Path Decomposer for Text-to-SQL

W4: Some technical introductions are unclear, which makes it difficult to understand the paper.

We provide detailed responses to your questions:

---

W4-1: What does the constraint or sub-SQL mean, and what can be considered as a constraint?

Constraint: In simple terms, a constraint refers to each operation expression in an SQL query. For example, in the SQL query "SELECT name FROM person WHERE age=18", components like "SELECT name", "FROM person", and "WHERE age =18" are all considered constraints. A detailed definition of constraints can be found in Appendix A.1 (the overall process of READER) and Figure 16 (constraints identified in the AST).

sub-SQL: A sub-SQL is an SQL query derived by removing one or more constraints from the original SQL query. For instance, in the above SQL, removing the constraint "WHERE age=18" results in "SELECT name FROM person", which is a sub-SQL. A more complex example is shown in Figure 8 (lines 2050).

---

W4-2: What are the goals of the Generator and Corrector, how are they used in SQL generation, and how is the Corrector trained and realized?

• The goal of the Generator is to minimize problematic constraints--specifically misinterpreted, omitted, and unwanted constraints--as illustrated in Fig. 1. This is achieved through a well-designed mechanism that effectively generates augmented data for further training.
• The goal of the Corrector is to refine low-confidence constraints in the Generator and reduce the errors of the Generator.

The reasoning process of the Generator and the Corrector is as addressed in A3 of our Global Comment. Secondly, the training of the Coeector includes the following steps:

• Construct the training set of the Corrector: We perform four-fold cross-validation on the BIRD and Spider training sets, respectively. We train the Generator on any three folds and perform reasoning on the remaining fold. According to the reasoning results, the training data of each fold is constructed, and finally combined to form the training sets of the Corrector on BIRD and Spider, respectively; see more details in Appendix A.4 (lines 1101-1108).
• Fine-tuning training: The training of the Corrector also uses LoRA fine-tuning with the default setting.

---

W4-3: The proposed method regenerated the question based on the sub-SQL, so what is the relation between the generated question and the original ones, and have the authors considered their consistency? Besides, how did the authors ensure that the sub-SQL could solve the generated question?

Figure 10(lines 2105) illustrates an example of the original (question, SQL) pair alongside two (sub-question, sub-SQL) pairs. It demonstrates that the generated questions differ from the original primarily due to difference in only one constraint.

To address the consistency issue, as shown in Figure 9(lines 2080), we utilize the original question pair as a demonstration for the LLM, guiding it to generate a question that corresponds to the sub-SQL. Our aim is to closely mimic the format of the original query to ensure consistency between the two. Additionally, we select high-quality generated data through a combination of rule-based criteria and manual screening, as detailed in lines 1023-1079 of the paper.

Thank you for your thoughtful feedback and thorough review. Below, we provide responses to your questions and concerns.

W1: The proposed seemed to be an incremental improvement of the existing Text-to-SQL methods, and the technical contributions seemed to be limited.

Thank you for the feedback. We argue that the contribution of READ-SQL lies not only in its exceptional performance but also in its potential to inspire mainstream adoption for real-world Text-to-SQL applications and to generate new ideas for code analysis in other languages. For further explanations, please refer to A1 of the Global Comment.

---

W2: The inference pipeline of the proposed framework is unclear. The paper detailed introduced the data generation process, but it did not show how the trained Generator and corrector were used to generate the SQL, i.e., the paper lacked an overall description from the text input to the SQL output.

Thank you for your valuable suggestion. We have incorporated your feedback into the revision and updated it in Sec. 3.4 (lines 304-323). For more details, please refer to A3 of the Global Comment.

---

W3: The motivation behind the proposed method should be clarified more clearly. For example, what is the purpose of constraint identification and decomposition, and why can it solve the problems mentioned in the introduction.

Thank you for your thoughtful feedback. The goal of READ-SQL is to bridge the gap between natural language and SQL within an end-to-end generation framework. We emphasize establishing connections between the constraints in the problem and the corresponding SQL clauses. We highlight the key steps of our READ-SQL to improve performance as follows:

• Enhancing the model's understanding of the nuances in SQL questions by incorporating additional question/SQL pairs for training;
• Providing reason to bridge the gap between the question and the SQL statement;
• Refining low-confidence clauses to further minimize the occurrence of erroneous constraints.

The implementation of these solutions relies on READER, which identifies and decomposes constraints, generates sub-SQL statements, and establishes reasoning paths.

We specifically examine the performance of READ-SQL and SFT CodeS in addressing the three types of errors mentioned in the introduction. The results are detailed in the revised paper (lines 514-521). READ-SQL effectively reduces the occurrence of these errors, particularly the phenomenon of omitted constraints, which has been decreased by nearly 48.7%.

W3: Although augmented data enhances performance, generating question/reason pairs involves complex processing that may pose implementation challenges. More importantly, even with additional training data, the performance gains on the Spider dataset are less significant than on the BIRD dataset. This raises the question of whether the observed improvements might be due to overfitting, as the paper indicates that most of the augmented data originates from the BIRD dataset.

Thank you for your insightful feedback. We respond to your questions as follows:

---

W3-1: There is a complex process involved in generating question/reason pairs.

A3-1: We emphasize that by executing READER on a SQL query, it can easily derive the corresponding clauses, sub-SQLs, and reasoning paths. We then use the LLM to generate (sub-question, sub-SQLs) and (question, reason) pairs, where reason is a description of the reasoning path of constructing the sub-SQLs. This procedure can be further explored by simply rule-based methods.

---

**W3-2: **Whether the observed improvements might be due to overfitting?

A3-2: We conduct a difficulty analysis and comparison between READ-SQL and SFT CodeS on three Spider variants. The results show that READ-SQL is ahead of SFT CodeS in EX indicators. To further explore the model's generalization, we have provided more detailed results in A2 of the Global Comment.

---

W3-3: The most of the augmented data originates from the BIRD dataset.

A3-3: Our enhanced data will be generated not only for the BIRD training set but also for the Spider training set.

---

Q1: What specific advancements does READER bring compared to earlier SQL decomposers?

READER is a self-developed method based on SQLglot. Compared with the existing SQL decomposition, it can divide SQL clauses more finely, split all possible executable sub-SQLs, and obtain multiple reasoning paths. We compare it with two recently published works, for example:

• STEPS $1$ : It cannot fine-grain SQL, resulting in clauses not being independent units.
• DeSQL $2$ : The SQL decomposer used has limited SQL structures involved, cannot handle nested queries, etc., and can only decompose simple SQL.

The following are detailed comparison results:

We show examples of these three methods in Appendix A.1.4 of the revised paper.

$1$ Yuan Tian, Zheng Zhang, Zheng Ning, Toby Jia-Jun Li, Jonathan K. Kummerfeld, and Tianyi Zhang. Interactive text-to-sql generation via editable step-by-step explanations. In EMNLP, pp. 16149–16166. Association for Computational Linguistics, 2023.

$2$ Sabaat Haroon, Chris Brown, and Muhammad Ali Gulzar. Desql: Interactive debugging of SQL in data-intensive scalable computing. Proc. ACM Softw. Eng., 1(FSE):767–788, 2024

---

Q2: How scalable is READ-SQL beyond 3 billion parameters, and are there plans for optimizing even larger-scale models?

READ-SQL can certainly work on larger-scale models. The reason for this is that we expand our training datasets with new question/SQL and question/reason pairs. The new question/SQL pairs enhance the model's sensitivity to questions, while the question/reason pairs improve its reasoning ability in end-to-end generation. Additionally, we use READER to enable the Corrector to re-evaluate low-confidence clauses, further boosting text-to-SQL accuracy.

However, due to our current resource limitations, we can only conduct experiments with models up to 3B. Nonetheless, we are confident that READ-SQL can scale effectively and deliver even better performance when applied to models with more parameters and augmented data.

---

Q3: Through augmented data generation, where question/SQL pairs and question/reason pairs are created, READ-SQL enhances comprehension of complex queries. Open-sourcing this related data would further support the broader community.

We commit to following ICLR's reproducibility guidelines and to making data, code, and runtime logs public.

Thank you for your thoughtful feedback and thorough review. Below, we provide responses to your questions and concerns.

W1: The paper highlights three key issues in text-to-SQL: misinterpreted constraints, omitted constraints, and unwanted constraints. However, these issues are neither addressed nor mentioned in the experimental section, nor are they resolved in the proposed approach, which I find confusing.

Thank you for your thoughtful feedback. We provide a more detailed analysis in the experimental section. Additionally, we emphasize that the goal of READ-SQL is to enhance the model's sensitivity to constraints in the question and to regenerate low-confidence constraints, thereby reducing the occurrence of these three issues. In the revised paper, lines 514-521, We add a relevant experiment to confirm this conclusion:

We present the frequency of the three types of errors occurring in two scenarios: cases where READ-SQL is correct but SFT-CodeS is wrong, and vice versa. It can be seen that READ-SQL has reduced the occurrence of all three error types, particularly the omission errors, which decreased by 48.7%.

---

W2: The authors state, “However, existing methods primarily focus on optimizing SQL generation without improving the model’s understanding of the question or establishing strong relationships between questions and SQL clauses.” From my perspective, schema linking already serves to establish relationships between queries and SQL structure, which weakens this point as a unique limitation. ... ...Therefore, this limitation also seems unconvincing. Overall, the motivation for READ-SQL remains unclear to me.

We answer the questions one-by-one below:

W2-1 : schema linking already serves to establish relationships between queries and SQL structure.

We highlights that while pattern linking helps the model establish connections between the problem and SQL, we still observe instances of misinterpreted constraints, omitted constraints, and unwanted constraints in the model's output under the schema linking scheme proposed by CodeS, a state-of-the-art method, the results in Appendix A.5.2 of the revised paper. Therefore, we hypothesize that these critical errors in SQL generation arise from the structural differences between natural language and SQL. Natural language is often vague and changeable, whereas SQL is structured programming language. Our research focuses on how to bridge this gap.

W2-2: READ-SQL does not appear to be significantly better than other methods in terms of understanding the question.

We specifically explore the performance of READ-SQL and SFT CodeS in the face of the three types of errors mentioned in the introduction. The results are shown in the revised paper at lines 514-521. READ-SQL effectively reduces the occurrence of the three situations, especially the occurrence of the omitted constraint phenomenon, which is reduced by nearly 48.7%.

W2-3: The motivation for READ-SQL is unclear

Just as W2-1 replied, bridging the gap between natural language and SQL is the focus of our work. In the Introduction section (lines 070-081), we outline two types of methods that use intermediate representations to address these differences: (1) SQL-like grammar languages and (2) direct output of SQL structures. However, these two types of methods do not support end-to-end SQL generation, which may result in the loss of critical information."

Differently, our READ-SQL provides an end-to-end generation solution. We emphasize establishing connections between the constraints in the problem and the corresponding SQL clauses. We highlight the key steps of our READ-SQL to improve performance as follows:

• Enhancing the model's understanding of the nuances in SQL questions by incorporating additional question/SQL pairs for training;
• Providing reason to bridge the gap between the question and the SQL statement;
• Refining low-confidence clauses to further minimize the occurrence of erroneous constraints.

The above three points need to be achieved with the help of our invention, READER, the SQL decomposer. Furthermore, our focus is not solely on the end-to-end generation of SQL, but on establishing the connection between the question and SQL within an end-to-end framework to bridge the gap between natural language and SQL.

Thank you for your thoughtful feedback and thorough review. Below, we provide responses to your questions and concerns.

W1: The main weakness of this paper lies in the experimental section. The baselines used are primarily methodologies that do not represent the most recent advancements in the field, such as Distillery, CHASE-SQL, and CHESS. Although the paper states that the main baseline is the SFT CodeS model itself, it would be beneficial to compare their approach with more advanced fine-tuning methods, such as using zero-shot chain-of-thought prompting to generate reasoning steps and training the CodeS model with those reasoning paths.

Thank you for your valuable feedback. In the revised paper, we have incorporated comparisons with the current state-of-the-art benchmarks, as shown in Table 1 and Table 2, while these benchmarks predominantly rely on closed-source models. We emphasize that our work focuses on small-scale models and their applicability in resource-constrained environments. In addition, we also experiment with the fine-tuning method you mentioned, using zero-shot chain-of-thought to generate reasoning. Below is a comparison of its performance with the reasoning path method we obtained through READER parsing:

Our method ensures that no SQL clauses are omitted, unlike zero-shot CoT, which may fail to retain clauses such as LIMIT 1. The prompts for zero-shot CoT are included in the supplementary materials, while the prompt used in our method is presented in Figure 12 of our paper.

---

W2: According to the results shown in Table 2, there are two instances where the performance of READ-SQL is lower than the SFT version of the CodeS model. This may indicate overfitting on the Spider dataset, which could explain why READ-SQL outperforms the SFT CodeS model on Spider specifically.

Thank you for your insightful observation. We conduct a difficulty analysis and comparison between READ-SQL and SFT CodeS on three Spider variants; see Appendix A.5.9 (lines 1323-1348) in the revision paper. The results show that READ-SQL is ahead of SFT CodeS in EX indicators. To further explore the model's generalization, we have provided more detailed explanations in A2 of the Global Comment.

---

W3: To better assess the effectiveness of their proposed correction methodology, it would be valuable to compare it with basic self-correction methods, such as those outlined in the DIN-SQL paper. This comparison would help determine whether the improvement comes solely from an additional LLM call or from the specific advantages of the proposed method.

We present the comparison between standard self-correction (GPT-4o) and Corrector (GPT-4o) in Table 4. The results confirm that the improvement in Corrector's capabilities is not solely attributed to GPT-4o.

In addition, we compare Corrector (GPT-4o) with the self-correction method in DIN-SQL, and the results are as follows:

Notably, DIN-SQL's self-correction method may lead to additional reasoning outputs, resulting in higher token consumption. The results demonstrate that our self-correction method significantly improves performance while incurring less API overhead. We show the prompts used by DIN-SQL and Corrector (GPT-4o) in Figures 13 and 14 of the revised paper.

Q1: Can READ-SQL work on larger-scale models (e.g., 13B, 34B LLMs)? Since the proposed framework already achieves comparable performance with small-scale LLMs, it would be interesting to explore whether the framework could outperform GPT-4-based methods when incorporating 70B models. Alternatively, if READ-SQL cannot work on larger-scale models, it would be helpful to explain why.

READ-SQL can certainly work on larger-scale models. The reason for this is that we expand our training datasets with new question/SQL and question/reason pairs. The new question/SQL pairs enhance the model's sensitivity to questions, while the question/reason pairs improve its reasoning ability in end-to-end generation. Additionally, we apply READER to enable the Corrector to re-evaluate low-confidence clauses, further boosting text-to-SQL accuracy.

However, due to our current resource limitations, we have not been able to experiment with these larger-scale models. Nonetheless, we are confident that READ-SQL can scale effectively and deliver even better performance when applied to models with more parameters.

---

Q2: Since the most direct comparison for the model at the same scale is with SFT CodeS-1B and SFT CodeS-3B, could the authors provide a detailed analysis of the specific types of questions where READ-SQL shows improvement? A statistical analysis of correct samples based on difficulty level, along with a straightforward case study comparing READ-SQL with CodeS, would help readers better understand the contributions and improvements.

Thank you for your thoughtful suggestion. We conduct a statistical analysis of the BIRD Dev, Spider Dev and Spider variants based on difficulty level (the results are added to the revised paper), as shown in Appendix A.5.9(lines 1323-1348). The following are the results on BIRD Dev:

Our findings in BIRD are as follows:

• The Generator generally outperforms the SFT-CodeS, showing the greatest improvement at moderate difficulty levels.
• With the introduction of the Corrector, there are improvements at both simple and moderate difficulty levels, though its effectiveness may decrease at more challenging difficulty levels.

The following are the results on Spider Dev:

Our findings in Spider Dev are as follows:

• The Generator exhibits the highest growth rate at hard difficulty.
• The addition of the Corrector does not lead to significant improvements. We believe this is because the Spider dataset, unlike the BIRD dataset, lacks additional external information. As a result, Corrector relies solely on column names to determine clause constraints.

In addition, we provide a case study with multiple samples showcasing a direct comparison between READ-SQL and SFT CodeS in revised paper as shown in Table 19 to 24 (lines 1366-1457).

Thank you for your thoughtful feedback and thorough review. Below, we provide responses to your questions and concerns.

**W1: The baseline included in the paper is insufficient. As a submission in October 2024, and in reference to recent studies $1$

$2$ , the paper should compare against more advanced baselines, such as $3$

$4$

$5$

$6$ . Although this comparison of performance may not be entirely fair due to the varying scale of model parameters, the authors could highlight their advantages in other aspects, such as model size, GPU resources, and runtime. Comparing to up-to-date baselines would strengthen the paper's position for submission to a top conference.**

Thank you for your valuable feedback. In the revised paper, we have incorporated comparisons with the current state-of-the-art benchmarks,as shown in Table 1 and Table 2, while these benchmarks predominantly rely on closed-source models. We emphasize that our work focuses on small-scale models and their applicability in resource-constrained environments. However, due to limited resources, we are unable to fully reproduce the larger open source model as well as the closed source model to gain running time and API overhead. We highlight model size as a metric in the revised paper.

---

W2: The paper lacks a detailed runtime analysis. Given the focus on using smaller-scale LLMs, adding an analysis of training and inference times would underscore the practical advantages of the lightweight design. Such an analysis would further emphasize the framework's efficiency in real-world settings, particularly for on-device and resource-constrained applications, thereby enhancing the paper's impact.

Thank you for valuable suggestions. In Appendix A.5.5 of the revised paper, we have updated it as follows:

We conduct a comparative runtime analysis of READ-SQL and SFT-CodeS on NVIDIA RTX 3090 GPUs with 24 GB of memory. The experiments are carried out using a two-card setup for training and a single-card setup for inference, the result as follows:

Among them, SFT CodeS adopts full fine-tuning, and the training time cannot be displayed due to resource constraints. READ-SQL comprises two components: the Generator and the Corrector. Data is presented in a sequence where the Generator precedes the Corrector. We get the following conclusion:

• READ-SQL offers an advantage of achieving high performance with minimal memory usage.
• Leveraging the LoRA fine-tuning method, it enables efficient training and inference in low-resource environments compared to full fine-tuning.
• Using only the Generator-3B improves performance by 1.96% compared to a single 3B model (SFT CodeS-3B), with slight increase in inference time (2.97s vs. 2.81s) due to additional computation on the LoRA. Remarkably, its performance is only 0.02% lower than that of SFT CodeS-7B.
• READ-SQL-3B surpasses SFT CodeS-7B by nearly 0.37% with smaller inference time and less memory usage.

---

W3&Q3: The model's generalization and its improvements on Spider datasets and their variants are not as strong as those observed with the BIRD dataset.

Thank you for your insightful questions. We conduct a difficulty analysis and comparison between READ-SQL and SFT CodeS on three spider variants. The results show that READ-SQL is ahead of SFT CodeS in EX indicators. To further explore the model's generalization, we have provided more detailed explanations in A2 of the Global Comment.

Thank you for your careful observation. We have corrected the presentation and spelling errors, as shown in the revised paper. Below, we address your questions and concerns.

W1: The writing could be improved to be more direct and accessible. Section 3.4 is incomplete — it only describes the extraction of common and different constraints without explaining the subsequent self-correction process. Some insights can be found in Figure 4, but the absence in the main text hinders comprehension.

Thank you for your valuable suggestion. We have incorporated your feedback into the revision and updated it in Sec. 3.4 of the reviewed version. For more details, please refer to A3 of the Global Comment.

---

W2: The comparisons with other models are somewhat unfair; READ-SQL-3B includes two 3B models (i.e., a 3B Generator and a 3B corrector), so a direct comparison with CodeS-3B may not be appropriate. &W3: The performance improvements are modest, especially considering the combined use of two 3B models.

We fine-tune CodeS-3B using LoRA to create a Generator adapter and a Corrector adapter, resulting in a minimal parameter increase of only 0.972% (0.486% per adapter), far from constituting two separate 3B models. Notably, READ-SQL surpasses the SFT CodeS-7B SOTA model on BIRD Dev, showcasing substantial improvement at the 3B scale. For a fair comparison, the results of individual Generators are detailed in A2 of the Global Comment.

---

Q1: Line 44 mentions three typical categories of text-to-SQL errors. Are there any statistics on the frequency of these error types?

In the revised paper, lines 1146-1150, we have updated it accordingly. We randomly select 100 error samples from the 690 total errors in SFT CodeS-3B and manually categorized them (Note: a single erroneous sample may contain multiple errors). The results are shown as follows:

---

Q2: How would the performance be affected if question-SQL pairs and question-reason pairs were combined into question-reason-SQL chain-of-thought data for fine-tuning CodeS ?

Thank you for your interesting suggestion. We have added additional experiments in lines 1206-1217 in the revised paper. We apply the Reason-SQL Formatting (RSF) method, as mentioned, setting the output format to REASON: {reason text};\nSQL: {SQL query}. We train on the BIRD training set and reason on BIRD Dev. The results are as follows:

This approach is clearly unsuitable for small language models (SLMs) for the following reasons:

• The reason output during model inference contributes to error accumulation, interfering with subsequent decoding and introducing unnecessary errors.
• Beam search is used for decoding, and including the reason output reduces the available space for the SQL portion, limiting the model's ability to generate diverse SQL queries.