Knapsack Schema Linking Agent for LLM-Based Text-to-SQL Generation

Thanks for your insightful suggestions and questions. We have addressed all additional discussions in our revision. We are eager to continue the discussion if you have further questions or concerns.

Q1: In the Guaranteed Scoring Model, the authors used the RoBERTa-Large model to obtain semantic embeddings. It would be helpful if they provided a rationale for choosing this specific model.

Thanks for raising this important discussion about the choice of language model for guaranteed scoring model. We address this from two perspectives:

• RoBERTa-Large offers better generalization and understanding of language nuances, which are crucial for schema linking tasks where capturing subtle semantic relationships is important. RoBERTa improves upon BERT by employing an enhanced and more extensive training process, resulting in improved text representation ability. In the field of LLMs, RoBERTa is also involved in many works that augment LLMs in various downstream tasks [7].
• RoBERTa-Large achieves the best performance compared to other language models in our setting. We compared different models and sizes of the guaranteed scoring model on SQL generation and found that RoBERTa-Large produced the best results on both datasets. Additionally, using fewer parameters (BERT-base or RoBERTa-base) or more parameters (SGPT-1.3B) did not lead to performance improvements. This indicates that RoBERTa-Large is a good choice for the guaranteed scoring model.

[7] Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

W2: The related work is severely lacking and is in the supplementary section instead of the main paper. The authors make it easy to understand what they have done but it is not clear what the previous approaches were and how exactly the proposed approach is better.

Thanks for your comments. In the revised manuscript, we have reorganize our paper to provide a more clear presentation of the proposed approach.

• We have added a new paragraph in the related work section of the paper to primarily discuss schema linking, which has been updated in the revision. In this paragraph, we examine the current progress in schema linking, including several recent works [2-6]. This discussion helps clarify recent developments in schema linking, highlight the challenges, and explain the impact of our work.
• The main impact of our KaSLA compared to baselines is that it addresses both element missing and element redundancy. Recent schema linking work often seeks to enhance performance by using in-context learning to boost reasoning [2, 6], employing larger LLMs [4], or augmenting contextual information [3, 5]. However, the current work does not emphasize the issues of missing elements and redundancy. Our KaSLA models schema linking as a knapsack optimization problem to reduce redundancy while ensuring no missing elements, better meeting the needs of SQL generation.

We have updated our related work section. However, due to page limitations, we may currently have to place the related work section in the appendix.

[2] Chess: Contextual harnessing for efficient sql synthesis

[3] E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

[4] The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models

[5] RSL-SQL: Robust Schema Linking in Text-to-SQL Generation

[6] CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL

Dear reviewer QByZ,

Thanks very much for your insightful suggestions and concerns. We have provided a detailed response to each concern and made improvements in the revised vision.

W1: Since this approach is based on dynamic programming, the computation time might be significant. It would be beneficial if the authors addressed this issue in the paper.

Thanks for your comments. The complexity of the dynamic programming (DP) process is very low since we utilize a hierarchical structure to select the schemas. Additionally, the running time analysis shows that the computational time of the DP process is minimal.

• KaSLA utilizes a hierarchical structure to reduce the time complexity: The time complexity of the dynamic programming component in KaSLA is $O(n _t C _t + n _t(n _c C _c))$. This relates to the database's table number $n _t$, column number $n _c$ of the table, and the estimated table capacity $C _t$, and column capacity $C _c$. For a traditional 0-1 knapsack problem with $n$ items and a capacity of $C$, its time complexity is $O(nC)$ [1]. In our hierarchical KaSLA strategy, we first optimize the table linking and then optimize the column linking within the linked tables. Thus, the total time complexity is $O(n _t C _t) + n _t O((n _c C _c)) = O(n _t C _t + n _t(n _c C _c))$.
• The actual time complexity $O(n _t C _t + n _t(n _c C _c))$ is small because the $n _t, n _t, C _t, C _c$ are usually small. We averaged these values for BIRD-dev and Spider-dev and displayed them in the table below. The results show that the actual time complexity is small, making our dynamic programming component time-efficient.

Table W1.a Average schema element number and average capacity of BIRD-dev and Spider-dev

• Inspired by your question, to understand the tradeoff between KaSLA's time consumption and performance, we provide the inference time on one 80G A100 GPU for the whole framework, both with and without KaSLA, in the table below. we found that the dynamic programming only takes less than 0.01 seconds to process one instance in both BIRD-dev and Spider-dev.

Table W1.b Inference time cost of each component in KaSLA

We also provide the SQL generation accuracy for text-to-SQL, both with and without KaSLA, as follows:

Table W1.c Text-to-SQL accuracy with full Schema and KaSLA

As shown in these tables, we can conclude that:

• KaSLA introduces a significant improvement with a marginal inference delay. This delay occurs because we use StarCoder2-15B to generate nomination scores for the full schema, aiming for better semantic understanding, which results in additional time consumption. Reducing the inference time for LLM-based nomination models during full schema processing might be a potential research direction.

[1] https://en.wikipedia.org/wiki/Knapsack_problem

Dear reviewer QByZ,

Thanks again for taking the time to review our paper and for providing valuable suggestions and concerns. We have carefully responded to each of your questions. Following your suggestion, we have provided a time complexity analysis of the dynamic programming component, explained the choice of RoBERTa-Large along with the corresponding ablation study, and updated the discussion on related schema linking work in the revision. We hope these responses can address your concerns and thanks for your support.

As the author-reviewer discussion deadline approaches, we would be very grateful if you could share your valuable thoughts with us. If you have any further concerns, we look forward to further discussion with you.

Thanks for your helpful suggestions and concerns. We've added the additional discussions to our revision. We welcome more discussions on any remaining issues.

Q2.11 And then, in L247, an importance value assigned by RoBERTa to each (surviving?) schema element. L271 "schemas with higher importance scores should be assigned higher values". Then we see Is,q which comes from eqn (4), combining inclusion by StarCoder2 and scoring by RoBERTa. This is confusing, because on the latter was called "imporance" earlier.

We have revised this section in the updated version to clarify any misunderstandings.

• As shown in Equation 4, our importance score $I_{s,q}$ combines the high confident score from StarCoder2 ($I_{nomi}$ ) and the general score from RoBERTa ($I_{guar}$). "Higher importance scores" refers to a higher $I_{s,q}$, instead of $I_{nomi}$ or $I_{guar}$.

Q2.12 L281 "schemas with higher importance should be assigned lower weights than those with lower importance scores". So the importance score figures twice: once in value, and again in weights. This seems not-elegant.

Thank you for your concern. We have reformatted the equations in our paper for improved clarity.

• For not merging $\hat{V} _{s,q}$, $\hat{W} _{s,q}$ and $I _{s,q}$, the reason is that we aim to treat the importance score estimation and the knapsack factor estimation (value, weight, and capacity) as two separate components. This makes the entire framework clearer and easier to develop, as the importance scoring model can be trained once and reused anytime.

Q2.13 L488 Rcorrect springs another surprise. Given the relatedness to recall and precision, why not just use F1?

• We addressed this concern in the response of W3, please refer to the above response.

Q2.14 Success in Table 3 is more mixed.

• Thanks for your feedback on the presentation of the experimental results. We have revised Table 3 in the updated version to better emphasize the most critical components.

Q2.8 One can measure a large value of your redundancy measure, but it is not clear how much that damages SQL generation (compared to loss of schema recall). This is worth a separate study.

• Thank you for suggesting this insightful approach. Investigating how variations in the redundancy measure affect SQL generation accuracy is indeed an enlightening idea. We have conducted related experiments, and the results are as follows. As shown in the table, adding redundant elements significantly negatively impacts text-to-SQL performance and increases the redundancy ratio.

Table Q2.a Randomly add 10%, 20%, and 30% redundant columns into all instances' schema linking results.

Q2.9 Eqn (2) is a measurement of loss of recall. "Redundancy rate" in eqn (3) is like one minus precision. It is confusing that redundancy is forced to zero if the recall is not perfect.

• We addressed this concern in the response of W3, please refer to the above response.

Q2.10.1 The terminology in "nomination-guaranteed" gets even more confusing. About $I_{\text{nomi}}(s|q,S)$: while you have defined in eqn (5) the quantity Lnomi, I could not find an expression for $I _{\text{nomi}}(s|q,S)$ nearby. It seems there is no use of this score downstream. It is just an indicator for what schema elements StarCoder2 will generate.

• Thanks for your concerns. $I_{nomi}(s \mid q,S) = 1$ if $s \in \widehat{S}_{nomi}$ and $I _{nomi}(s \mid q,S) = 0$ if $s \notin \widehat{S} _{nomi}$, indicating whether each schema element $s$ appears in the generation results $\widehat{S} _{nomi}$ of the StarCoder2-based nomination model. Actually, $I _{\text{nomi}}(s|q,S)$ is used in Eq. (4), $I({s \mid q,S}) = I _{nomi}{(s \mid q,S)} + \alpha I _{guar}{(s \mid q,S)}$, as a part of the important score.

Q2.10.2 Authors need to provide exact input and output specifications for finetuning StarCoder2.

Below are examples of inputs and outputs for starcoder2-based nomination scoring model:

• Input of starcoder2-based nomination scoring model

database schema:
table department, columns = [ department_id (int|primary key|values: 1) , name (text|values: State, Treasury), ranking (int|values: 1, 2)]
table head ,columns = [ head_id (int|primary key|values: 1) , name (text|values: Tiger Woods) , age (real|values: 67.0)]
table management, columns = [ department_id ( int|primary key|values: 2), head_id (int|primary key|values: 5), temporary_acting (text|values: Yes)]
foreign keys: management.head_id = head.head_id, management.department_id = department.department_id
Question:
How many heads of the departments are older than 56 ? List the relevant columns in each table:

• Output of starcoder2-based nomination scoring model

"Relevant columns": {"department": null, "head": ["age"], "management": null}

• Turn the output to the nomination score $I_{\text{nomi}}(s|q,S)$:

{"department.department_id":0, "department.name":0, "department.ranking":0, "head.head_id":0, "head.name":0, "head.age":1, "management.department_id":0, "management.head_id":0, "management.temporary_acting":0}

Thanks for your suggestions, we have updated the details about the input and output of StarCoder2 in our revision.

Q2.10.3 Also in eqn (5) how do you deal with the fact that S may be output in any order?

• Thanks for your question. We handle the fact that $S$ may be output in any order by aligning the order of elements in the linking results with those in the full schema input. Our approach allows StarCoder2 to scan the full schema from beginning to end, selecting matching elements and placing them in the output in the same order. By aligning the output order with the input order, the absence of a specific order in schema linking does not affect the training process.

Q1 https://bird-bench.github.io/ shows systems with better performance, but perhaps the authors are interested in a more controlled comparison? They should explain their position in this matter.

Thank you for referencing the BIRD benchmark, which highlights some of the best performances in the text-to-SQL field. We appreciate to discuss KaSLA's position concerning these strong performances, and we address this step by step:

• Most of the top-ranking do not release their code, and we cannot reproduce their results. For methods where the paper is available but the code is not, we found it difficult to achieve results similar to those reported by relying solely on the details provided in their papers. Therefore, in our experiments, we primarily considered methods that have both released papers and available code.
• We compared CHESS (top 6) and E-SQL (top 15) and found that KaSLA can enhance them. They reported results on BIRD-dev that exceed those of our KaSLA with the StarCoder2-15B-based SQL generation model. Since KaSLA is a plug-and-play schema linking model, we reproduced E-SQL and CHESS and compared their performance with and without KaSLA. As shown in the table below, integrating KaSLA can further enhance them by improving their schema linking, highlighting KaSLA's plug-and-play nature and robust schema linking capability.
• Implement details: CHESS [1] uses a retrieval-augmented chain-of-thought prompting method for schema linking, while E-SQL [2] employs a filtered schema correction strategy. We enhanced their schema linking methods with KaSLA. Since CHESS uses GPT-4 and a fine-tuned DeepSeek Coder model but did not release the latter's checkpoint, we reproduced each component in CHESS using GPT-4. In line with [2], we used GPT-4o as the base LLM for E-SQL.

Table Q1.a Text-to-SQL performance of integrating KaSLA into SOTA baselines (CHESS and E-SQL)

[1] CHESS: Contextual Harnessing for Efficient SQL Synthesis

[2] E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

W4 While some improvements result, Knapsack may not be technically the correct paradigm to model schema selection.

Thanks for raising this important question. We address it from two angles: the motivation for formulating schema linking as a knapsack problem and the necessity of a capacity constraint.

• Schema linking requires no missing elements and low redundant elements. Any missing schema element can cause SQL generation to fail directly, while redundant elements can confuse the SQL generation process and lead to incorrect outputs. As shown in the table, missing tables or columns and adding redundant ones significantly negatively impact text-to-SQL performance.
• These requirements align with the knapsack optimization goal of maximizing value while minimizing weight within a capacity constraint. Therefore, we formulate it as a knapsack problem to address both requirements.

Table W4.a Randomly drop 1, 2, 3 columns from all instances' schema linking results.

Table W4.b Randomly add 10%, 20%, and 30% redundant columns into all instances' schema linking results.

• KaSLA achieves optimal performance by attaining both the lowest column missing ratio and the lowest column redundancy ratio, demonstrating the effectiveness of the proposed knapsack optimization framework.

Table W4.c Performance on Bird-dev with execution accuracy (EX) for text-to-SQL evaluation and missing ratio and redundancy ratio for schema linking evaluation. The SQL generation model is a fine-tuned StarCoder2-15B

Our paper aims to introduce the idea that schema linking inherently possesses the characteristics of optimization problems, aiming for an ideal goal under constraints, particularly related to the classic and straightforward knapsack framework. Future research may explore alternative combinatorial optimization methods for schema linking.

W5 (L364-L373) are confusing. Why should the SQL generator get access to "ground truth schema S" at all? You would not be able to use it in a real deployment, would you?

Thank you for pointing this out. we have modified this section and updated a revision to address any confusion caused by wording.

• The ground truth schema $S$ is used only for training. In lines 364-373 of the submission version, we discuss the training of the SQL generation model, where the input schema elements combine KaSLA’s linking results and the training dataset’s ground truth linking results $S$. This ensures that the inputs contain all matching elements, as even the optimized KaSLA might not achieve 100% correct schema linking.
• During inference on the dev dataset (i.e., real deployment), the input schema elements for the SQL generation model are solely KaSLA’s linking results. No ground truth information from the dev dataset is available. To avoid confusion, we have added the inference process after the training process to explain the KaSLA deployment process more clearly.

(3) The special design of redundancy ratio considering no element missing

• Redundancy is deemed a failure if elements are missing because a prediction with missing elements is meaningless for redundancy evaluation, as explained in point (2).

(4) Why AUC is not suitable for schema linking

• AUC is unsuitable because schema linking is an imbalanced classification task. As shown in the table below, matching elements are only a small subset of all elements. A schema linking model can achieve a high AUC score by predominantly predicting non-matching elements and ignoring matching elements.

Table W3.e The average number of all element / matching elements of each instance in BIRD-dev and Spider-dev

(5) Why use $R_{correct} = 1 - (R_{miss} + R_{redun})/2$ instead of $R_{F1} = \frac{2 R_{recall} R_{precision}}{R_{recall} + R_{precision}}$

• $R_{correct}$ in our paper is designed to reflect the combined ability of a schema linking model to accurately predict matching elements while reducing redundancy. As shown in Table 3 of our paper, KaSLA achieves both a low missing rate and low redundancy rate, resulting in the highest $R_{correct}$.
• We didn't use F1 score, because recall and precision themselves are not suitable metrics, as explained above in point (1) and point (2).

We have updated the comparation between the traditional metrics with the proposed schema missing rate and schema redundancy rate in the revision.

(2) Necessary to propose missing ratio and redundancy ratio

Schema missing ratio: $R _{miss} =\frac{1}{|B|} \sum _{(q,S) \in B} \unicode{x1D7D9}(S _q \nsubseteq \widehat{S} _q)$

schema redundancy ratio: $R _{redun} =\frac{1}{|B|} \sum _{(q,S) \in B} \underbrace{{(\frac{|\hat{S} _q \setminus S _q|}{|\hat{S} _q|})}^{\sigma}} _{\text{redundancy ratio}}, \underbrace{\sigma = \unicode{x1D7D9}(S _q \subseteq \widehat{S} _q)} _{\text{Non-missing Indicator}}$

Due to the above drawbacks, precision and recall cannot be used in schema linking evaluation. Then we propose the missing and redundancy ratios to evaluate the schema linking accurately.

• Our schema missing ratio can reflect the actual impact of schema linking results to SQL generation by a strict evaluation. It considers an instance a failure if any element is missing, acknowledging success only when no elements are missing. This aligns with the fact that any missing element leads to incorrect SQL generation.
• In Table W3.a, the two examples show a missing ratio of 60% and 100% (lower is better), accurately reflecting the difficulty in generating correct SQL from these schema linking results.
• Lower missing ratio will have higher SQL generation accuracy: The table below shows the column missing ratio and SQL generation accuracy using the same text-to-SQL model with different schema linking models. The results demonstrate that the missing ratio accurately reflects the accuracy of schema linking and directly affects SQL generation. KaSLA achieves the highest SQL generation accuracy with the lowest missing ratio.

Table W3.c Performance on Bird-dev with execution accuracy (EX) for text-to-SQL evaluation and missing ratio for schema linking evaluation. The SQL generation model is a fine-tuned StarCoder2-15B

• Our schema redundancy ratio can provide a meaningful evaluation by calculating redundancy only when all matching elements are present in the linking results. The main difference between precision and our redundancy ratio is that the prediction with an element missing will be judged as a failure because, in such cases, redundancy is not the sole reason for incorrect SQL generation, thus making its evaluation meaningless.
• In Table W4.b3, the two examples show redundancy ratios of 100% and 20% (lower is better), correctly reflecting that the first schema linking result struggles to generate correct SQL while the second one can.
• Lower redundancy and missing ratios will have higher SQL generation accuracy. Using the same experimental setup, the table below shows the column redundancy ratio and corresponding SQL generation accuracy. The results in Tables W4.c and W4.d indicate that only KaSLA achieves the lowest redundancy and missing ratios, resulting in the highest accuracy of SQL generation.

Table W3.d Performance on Bird-dev with execution accuracy (EX) for text-to-SQL evaluation and missing ratio for schema linking evaluation. The SQL generation model is a fine-tuned StarCoder2-15B

W3 Instead of inventing new retrieval measures, why not not stick to classic ones like recall and precision? I do not quite buy that you have to "propose two metrics that evaluate the missing rate and the redundancy rate that directly impact the SQL generation performance, rather than indirect metrics like AUC and F1" Recall and precision directly impact SQL generation as much as the measures you propose.

Q2.9 Eqn (2) is a measurement of loss of recall. "Redundancy rate" in eqn (3) is like one minus precision. It is confusing that redundancy is forced to zero if the recall is not perfect.

Q2.13 L488 Rcorrect springs another surprise. Given the relatedness to recall and precision, why not just use F1?

Thanks for bringing up these insightful discussions about schema linking evaluation. We addressed these concerns by explaining the limitations of classic recall and precision and the advantages of the proposed schema missing rate and schema redundancy rate. We also explained why AUC and F1 are not suitable for schema linking.

(1) Limitations of precision and recall in Schema

Precision: $R_{precision}=\frac{1}{|B|} \sum _{(q,S) \in B}\frac{|\hat{S} _q \cap S _q|}{|\hat{S} _q|}$

Recall: $R_{recall} = \frac{1}{|B|} \sum _{(q,S) \in B} \frac{|\hat{S} _q \cap S _q|}{|S _q|}$

($B$ is the whole dev dataset, $\hat{S}_q$ is the predicted schema linking results, and $S_q$ is the ground truth.)

They are both general evaluation metrics; however, they have drawbacks that hinder their usage in schema linking.

• The same recall metric values can correspond to different SQL generation accuracies: In schema linking, missing a single table or column (recall < 100%) can significantly decrease text-to-SQL generation accuracy. Averaging recall cannot indicate how many instances are correct. We provide two examples in the table below.
• As shown in the table, in example-1, the average recall is 80%. However, since 3 out of 5 instances have missing elements, only 2 instances can generate correct SQL. In example-2, the average recall is also 80%, but with missing elements in every instance, not a single instance can generate correct SQL. This demonstrates that average recall does not accurately represent actual schema linking results.

Table W3.a Two examples of using recall as a column linking metric for 5 instances—high average recall does not equal high correct linking

• Higher precision metric values may result in lower SQL generation accuracy: Precision is not decisive in schema linking tasks. For instance, if the precision is 100% but the recall is below 100%, there will be 0 correct SQL generation. We provide two examples in the table below.
• As seen in the table, in example-3, all 5 instances have no redundant elements in their linking results, yet all instances still have missing elements. Precision is 100%, but no correct SQL can be generated. In example-4, there are no missing elements, but redundancy exists; therefore, precision is low, but all instances can potentially generate correct SQL. This indicates that precision does not clearly correlate with the success of schema linking results.

Table W3.b Two examples of using precision and recall as column linking metrics for 5 instances—precision cannot reflect linking results accurately

Dear reviewer dUxV,

Thanks very much for these insightful suggestions and concerns. We will respond to these suggestions and concerns in detail one by one.

W1 Proposed method has too many moving parts and lacks clarity in its development.

Thanks for highlighting this concern. We will explain the motivation for formulating schema linking as a knapsack problem and the development of KaSLA.

• Schema linking requires no missing elements and low redundant elements. Any missing schema element can cause SQL generation to fail directly, while redundant elements can confuse the SQL generation process and lead to incorrect outputs. As shown in the table, missing tables or columns and adding redundant ones significantly negatively impact text-to-SQL performance.

Table W1.a Randomly drop 1, 2, and 3 columns from all instances' schema linking results.

Table W1.b Randomly add 10%, 20%, and 30% redundant columns into all instances' schema linking results.

• The knapsack problem—maximizing value while minimizing weight within a capacity constraint—aligns well with schema linking by addressing both the missing and redundant requirements. Therefore, we formulate schema linking as a knapsack problem and propose KaSLA. The components of KaSLA align closely with knapsack optimization, incorporating factors such as value, weight, and capacity, along with a dynamic programming-based hierarchical schema linking mechanism. We apply importance scoring estimation to adapt knapsack optimization for schema linking.
• The table below shows that KaSLA achieves optimal performance by attaining both the lowest column missing ratio and the lowest column redundancy ratio, demonstrating the effectiveness of the proposed knapsack optimization framework.

Table W1.c Performance on Bird-dev with execution accuracy (EX) for text-to-SQL evaluation and missing ratio and redundancy ratio for schema linking evaluation

• We have released all the code to facilitate development and reproduction: the datasets used, the training and inference pipeline for importance scoring models, and the implementation of hierarchical knapsack optimization for schema linking. We have systematically organized the code for better usability. In practical deployment or future research, more focus can be placed on adapting estimation methods for importance scoring and knapsack factors.

W2 Writeup quality can be greatly improved (see below). Nonstandard and unfamiliar terminology is frequently used.

Thanks for the constructive feedback on the quality of the writing and terminology. We have addressed each sub-question in Q2 and incorporated your suggestions into our revision.

Q2 Many writing nits:

• Thanks for pointing out these suggestions to enhance the quality of our paper. We have carefully considered each of your suggestions and updated a revision . Below are the specific changes we have addressed:

Q2.1 What is described as "schema" throughout the paper is more accurately called "schema element", which is usually a table (including views) or a (table, column) pair in case of text2sql.

• We have made adjustments throughout the paper to ensure that "schema element" is used where appropriate to distinguish it from the full database schema.

Q2.2 I do not find it appropriate or necessary to call the proposed method an "agent".

• We refer to our model as an "agent" because its plug-and-play nature. Our KaSLA can be integrated into any text-to-SQL framework as a schema linking agent or serve as an enhancement to existing schema linking methods.

Q2.3 "Nomination-guaranteed" is not a thing. You cannot use this in an abstract (or anywhere, actually).

• Thank you for highlighting this. As you noted, using non-standard terminology in the abstract can cause confusion. We have revised the abstract accordingly.
• Within the main text, we describe the StarCoder2-based scoring model as a "Nomination model" because it typically generates binary score $I_{\text{nomi}}$ with high-confidence, akin to nominating select a few candidates from a pool. The RoBERTa-based scoring model is termed the "guaranteed model" since it provides a soft score $I_{\text{guar}}$ for each element, offering only a general representation.

Q2.4 (L307-L315) Why is γ needed as a hyperparameter?

• We employ $\gamma$ in the estimation function of capacity: $\widehat{C} _q = f _{C}(q) = \gamma \cdot \max {\lbrace \sum _{s \in \text{Knap}(d)} \widehat{W} _{s,q} \rbrace} _{d \in D}$, $\gamma>0$, to adjust the scaling of the estimated capacity. In practical applications, scaling to a larger capacity can include more potential matching elements in the linking results, but it may also overly relax constraints, leading to redundancy. We use $\gamma$ to manage this trade-off.
• We simply apply $\gamma = 1$ in our experiments, that is, use the normal capacity scale. we provide the Execution Accuracy (EX) results with $\gamma = \in \{0.5, 1, 1.5, 2\}$ in table below. The results show that $\gamma$ can slightly affect KaSLA's performance and offer the potential for the most optimal results. This demonstrates the generalization ability of KaSLA.

Table Q2.4.a Ablation study of $\gamma$

Q2.5 Various other terms are used throughout in confusing manners: redundancy, "semantically related schema".

Q2.6 L017 …but can be modeled as a Knapsack… L019 "redundant" or "incorrect", "excessive"? L023 "nomination-guaranteed" is non-standard. Define or use a more self-contained expression. L025 better to specify early that "capacity" is the maximum item weight the knapsack can support. This is found on L139 "weight capacity".

Thanks for highlighting these terms. We have revised the manuscript to ensure these terms are used consistently and clearly. We have made the following revisions in the revision to prevent any confusion:

• Use "matching element" and "un-matching element" to describe the linking relationship between schema elements and the query. Use "missing element" and "redundant element" to identify errors in the linking results. Remove unclear terms like "semantically related schema," "incorrect," and "excessive."
• Move the first mention of "capacity" to the start of the method explanation to clarify that "capacity" means the maximum weight the knapsack can hold.
• Add detailed definitions for the "nomination scoring model" and "guaranteed scoring model" to clear up any confusion caused by these names.

Q2.7 Figure 2 What are the schema items S1,…,S4? Better to call them "schema items" or "schema elements" and give actual examples of tables and columns.

• Thanks for your suggestion. Using "schema elements" instead of "S1,…,S4" indeed helps present our methodology more clearly. We used "S1,…,S4" due to the space constraints in the submission version.
• Following your recommendation, we have redrawn Figure 2 to use "schema elements" and include actual examples of tables and columns, enriching the figure's informative value.

I buy the argument that $R_{miss}$ and $R_{redun}$ are better tailored to schema retrieval than recall and precision, but that still does not explain why their combination cannot be via some form of harmonic mean. But I understand that once you justify moving away from recall and precision, there is less compulsion to restrict to standard combinations as well.

Basically an ablation replacing your measures with recall and precision. Does that make sense?

Thanks for your question. We conducted an ablation study replacing $R_{miss}$ and $R_{redun}$ with $R _{recall}$ and $R _{precision}$ during evaluation, and the results are shown in the table below.

From the comparison between $R _{recall}$ and $R _{miss}$ in Table 1, we can observe the following:

• $R _{recall}$ does not consistently correspond to SQL generation accuracy. Since missing elements directly lead to the inability to generate correct SQL, which recall does not accurately capture.

  • For example, DTS-SL ($R_{recall}$ = 77.73% ) and TA-SL ($R_{recall}$ = 77.74%) have nearly the same recall values, yet there are significant differences in their SQL generation accuracy.
  • KaSLA shows significantly better SQL generation performance than CodeS-SL. However, it is difficult to distinguish between them based on recall ($R_{recall}$ = 97.71% and $R_{recall}$ = 96.43%).

• $R_{miss}$ can reflect actual SQL generation accuracy because $R_{miss}$ provides strict constraints for any element missing.

  • The evaluation of DTS-SL and TA-SL using $R_{miss}$ is more reasonable. DTS-SL ($R_{miss}$ = 35.77%) performs better than TA-SL ($R_{miss}$ = 43.80%) because it has fewer instances with missing elements.
  • Using $R _{miss}$, the distinction between KaSLA and CodeS-SL becomes clear, with $R _{miss}$ = 5.83% for KaSLA and $R_ {miss}$ = 12.53 % for CodeS-SL.

Table 1. Evaluation with $R _{recall}$ 和 $R _{miss}$ on Bird-dev for different schema linking models with CodeS-15B as the text-to-SQL model

From the comparison between $R_{precision}$ and $R _{redun}$ in Table 2, we can observe the following:

• $R_{precision}$ is biased towards models with a severe element missing, which can mislead the evaluation because missing elements directly result in the inability to generate correct SQL.
  • For example, Pure-SL ($R_{miss}$ = 57.54%) and TA-SL ($R_{miss}$ = 43.80%) have significant element missing, yet they can easily achieve very high $R_{precision}$ scores ($R_{precision}$ = 80.06% and $R_{precision}$ = 84.92%, respectively), which is inconsistent with their SQL generation accuracy.
• $R_{redun}$ ensures that all necessary elements are included in the linking results when evaluating redundancy, thus preventing disruptions in accurate redundancy assessment.
  • By considering element missing, the evaluations by $R _{redun}$ for Pure-SL ($R _{redun}$ = 61.75 %) and TA-SL ($R _{redun}$ = 46.25 %) are more reasonable.
  • KaSLA achieved the lowest $R_{redun}$ = 42.24% and the lowest $R_{miss}$ = 5.83%, resulting in the best SQL generation accuracy compared to other methods.

Table 2. Evaluation with $R _{precision}$ 和 $R _{redun}$ on Bird-dev for different schema linking models with the same SQL generation model

Thanks very much for raising further concerns and discussing them with us. We appreciate the opportunity to speak with you, and if you have other concerns, we can discuss them in depth.

Dear reviewer dUxV,

Thanks again for taking the time to review our paper and for providing valuable feedback on our responses. We have carefully responded to your latest concerns. We discussed the intriguing idea of a metrics-based knapsack-like selection strategy thoroughly and provided an ablation study on traditional metrics, such as recall and precision, following your suggestions. We hope these responses can address your latest concerns and thanks for your support.

As the author-reviewer discussion deadline approaches, we would be very grateful if you could share your valuable thoughts with us. If you have any further concerns, we look forward to further discussion with you.

Thanks for your valuable feedback and concerns. We have addressed all additional discussions in our revision. We invite further discussion on any unresolved concerns.

W4. The paper lacks of details on the table representation. How is cell values considered?

Thank you for raising this important point.

• The inputs for both the StarCoder2-based nomination scoring model and the text-to-SQL model include cell values. The reasoning ability of LLMs can help us to understand these cell values to achieve better nomination scores and SQL generation. We use the retrieval method from [1] to obtain the most representative cell value for each schema element, treating it as part of the element's description.
• Below is the input format of the StarCoder2-based nomination scoring model. Each column has its corresponding cell value.

database schema:
table department, columns = [ department_id (int|primary key|values: 1) , name (text|values: State, Treasury), ranking (int|values: 1)]
table head ,columns = [ head_id (int|primary key|values: 1) , name (text|values: Tiger Woods) , age (real|values: 67.0)]
table management, columns = [ department_id ( int|primary key|values: 2), head_id (int|primary key|values: 5), temporary_acting (text|values: Yes)]
foreign keys: management.head_id = head.head_id, management.department_id = department.department_id
Question:
How many heads of the departments are older than 56 ? List the relevant columns in each table:

[1] CodeS: Towards Building Open-source Language Models for Text-to-SQL

W5. In real life application, table names and/or column names can be explicitly mentioned in the question. This motivates the use of sparse index such as inverted index. Can inverted index be used as a part of this approach to improve the recall/precision?

Thanks for your insightful suggestions. We conducted experiments to evaluate the impact of the sparse index method (BM25 + inverted index) on the schema linking task.

• We used the sparse index retrieval method for preprocessing the full schema for KaSLA. First, we collected schema element names and query words from the training dataset to construct a retrieval corpus mapping natural language words to schema element names. During inference on the dev dataset, we retrieved schema elements based on query words. For each schema entity, we calculated its score using BM25 and marked the top-1 element as the retrieved element. We then combined the full schema with the retrieved elements as inputs for KaSLA.
• Our experiments used the presented BIRD-dev data (843 instances with 6 databases not present in the training data) and the corresponding non-presented BIRD-dev data, as mentioned in our response to W1. The results are shown in the table below. They indicate that the sparse index method did not enhance KaSLA, possibly because (1) although public text-to-SQL datasets are quite large, they are still too small for constructing an effective retrieval corpus, leading to suboptimal retrieval results with noise, which affects KaSLA's performance, and (2) KaSLA's StarCoder2-based nomination scoring model and RoBERTa-based guaranteed scoring model can effectively capture schema element names included in the questions through semantic information, making the sparse index introduction unnecessary for improvement.

Thanks for proposing the inspiring idea of using the sparse index retrieval method in the text-to-SQL field. Although current text-to-SQL datasets may not yet support the construction of an effective retrieval corpus, limiting this idea's development, we still believe it is a very promising future research direction.

Table W5.a: Text-to-SQL Performance of KaSLA with sparse index on presented and non-presented dev data of BIRD-dev.

W2. The proposed method requires a fair amount of fine-tuning efforts. The application scenario is unclear. Such an approach can be valid for competing in benchmark like BIRD where the dev and holdout test has similar distribution. It works when the type of questions and the schema design in the database is fairly stable over time. In the event of significant schema changes, does the training/fine-tuning need to happen again? The paper lacks of discussion on the exact text-to-SQL usecase.

Thank you for bringing up this important question. We discussed the application of KaSLA in an exact text-to-SQL use case and found that the KaSLA trained on the public dataset can be directly applied to a new text-to-SQL scenario without additional training/finetuning.

• KaSLA trained on Spider shows strong performance on BIRD, and vice versa. As shown in the table, even when trained on other datasets, KaSLA provides good schema linking performance.

Table W2.a Text-to-SQL performance of KaSLA trained on Spider-train but evaluated on Bird-dev

Table W2.b Text-to-SQL performance of KaSLA trained on BIRD-train but evaluated on Spider-dev

• BIRD already considers transferability evaluation. As shown in the table, there are six dev databases with 843 instances not included in the training dataset. KaSLA performs well on both presented and non-presented dev data, demonstrating strong generalizability and transferability (as discussed in our response to W.1).

Table W2.c Text-to-SQL performance of KaSLA on the presented dev data and nonpresented dev data of BIRD-dev

W3. Text-to-SQL generation often deals with SQLs involving multiple tables (e.g. JOIN). Join keys can be semantically meaningless to the question (arbitrary id columns) but very important to the generation. It is unclear how is this taken into consideration when the proposed approach is performing column retrieval/selection.

Thanks for highlighting the challenge of handling multi-table joins. We will describe how our framework addresses column selection in scenarios involving multiple tables (e.g., JOIN).

• KaSLA addresses multi-table join tasks by utlizing attention networks in two importance scoring models. It employs a StarCoder2-based nomination scoring model and a RoBERTa-based guaranteed scoring model. During both training and inference, these models process all tables and their columns simultaneously. The attention mechanisms facilitate the connection of columns across different tables, enabling accurate linking when a query requires join operations.
• The results from instances involving JOIN operations in BIRD-dev demonstrate KaSLA's advantages over the baseline.

Table W3.a Text-to-SQL performance of KaSLA on the BIRD-dev data involving and not involving the JOIN operation

Dear reviewer jitH,

Thanks for your thoughtful suggestions and concerns. We have addressed each of your points carefully to alleviate any concerns and updated a revision.

W1. As mentioned in S1, as the number of tables in a database where the text-to-SQL is performed against is growing, the problem of table retrieval becomes increasingly challenging. To really assess the impact of this work, it will be great if the paper can discuss and address the size of full schema |S| being realistically large.

Thanks for noting KaSLA's advantage in large-scale real applications. LLM-based text-to-SQL faces the following challenges: 1) super large schema sizes; 2) human-defined table or column names.

1) Super large schema size

• KaSLA uses a hierarchical approach to shrink very super schema sizes. The results below demonstrate that for large-sized schemas (with column numbers ≥ 50 or table numbers ≥ 8), KaSLA shows more significant improvements compared to baselines than for smaller schemas (with column numbers < 50 or table numbers < 8). Especially when there are many columns, KaSLA has a more pronounced advantage.
• The main reason is that KaSLA employs a hierarchical process for schema linking, first linking tables and then linking columns within those tables. Compared to linking columns from all tables, this method is optimal because it reduces the selection space for columns, lowers time costs, and avoids noise and confusion from unlinked tables' columns.

Table W1.a Text-to-SQL performance on the BIRD-dev data with different table size

Table W1.b Text-to-SQL performance on the BIRD-dev data with different column size

2) Complex human-defined table or column names

• KaSLA is adaptive to complex human-defined table and column names because the nomination and guaranteed scoring models provide robust score estimates. These two scoring models deliver discriminative and generalized predictions to achieve robust importance score predictions, which is more robust in handling complex human-defined table or column names, enabling KaSLA to understand complex semantic information in real-world scenarios.
• In BIRD-dev, there are 6 databases (843 instances) with scenarios not present in the training data, meaning the semantic background of their table and column names was not covered during training. As shown in the table, KaSLA demonstrates a clear advantage over baselines in dealing with both presented and non-presented development data.

Table W1.c Text-to-SQL performance of KaSLA on the presented dev data and nonpresented dev data of BIRD-dev

Dear reviewer jitH,

Thanks again for reviewing our work, and a special thanks for your constructive questions and suggestions on applying KaSLA in real-world scenarios. We have provided detailed responses to each of your concerns, emphasizing the challenges of the schema linking task, supplementing with experiments on cross-scenario transfer capability, and discussing the integration with index retrieval methods, among other detailed responses. We hope these responses can address your concerns and thanks for your support.

As the author-reviewer discussion deadline approaches, we would be very grateful if you could share your valuable thoughts with us. If you have any further concerns, we look forward to further discussion with you.

Thanks for your insightful suggestions and concerns. We have incorporated all additional discussions into our revision. We are also open to further discussions on any remaining concerns.

Implementation details:

• KaSLA has good generalizability and does not require careful hyperparameter tuning. We simply set these parameters as: $\alpha = 1$, $\tau = 0.5$, $\gamma = 1$ in the paper.
• We also provided results considering a range of these hyperparameters in the table below. The results show that the hyperparameter values only slightly affect KaSLA's performance and offer the potential for the most optimal results. This demonstrates the generalization ability of KaSLA.

Table W2.3.a Ablation study of hyperparameters

W3.1 Be more careful about the notation details, e.g., in Eq.(8), fV(s), fV(s,q), and EI≥τ(S), if the result is related to concrete input question q, please add the symbol q. Otherwise, the reviewers will wonder whether this is a benchmark-level statistics or instance-level score.

Thanks for identifying this issue. We have refined the equations and notations in the revision.

• For $\mathbb{E} _{I \geq \tau}(S)$, since its inputs should involve $q$, we have updated its notation from $\mathbb{E} _{I \geq \tau}(S)$ to $\mathbb{E} _{I \geq \tau}(S, q)$ in the revised version. Since $f _{V}(s,q)$, $f _{W}(s,q)$, and $f _{C}(q)$ are function definitions, their format remains stable, with $q$ and $s$ as inputs. Therefore, we maintain their format in the revision.

W3.2 Try to simplify some equations to avoid over-complicating the introduction. For example, the value estimation Vs,q re-uses the importance score in Eq.(7), why not merge them into one? This will also alleviate issue i).

Thanks for your concern. We have reformatted the equations in our paper for improved clarity.

• For not merging $\hat{V} _{s,q}$ and $I _{s,q}$, the reason is that we aim to treat the importance score estimation and the knapsack factor estimation (value, weight, and capacity) as two separate components. This makes the entire framework clearer and easier to develop, as the importance scoring model can be trained once and reused anytime. While we use a simple positive correlation for value estimation $\widehat{V} _{s,q} = f _{V}(s,q) = I _{s,q}$, we encourage further exploration of optimized estimation methods.

(2) Eq. (10)

For query $q _d$ in the training data, we use ${S^{'}} _d$ to denote its ground truth schema linking results. We use different symbols ${S^{'}} _d$ here compared to our paper because markdown cannot display the original symbols.

Eq. (10): $\text{Knap}(d) = {S^{'}} _d + { \lbrace s \mid I _{s,q _d} \geq \min { \lbrace I _{z,q _d}\rbrace } _{z \in {S^{'}} _d} \rbrace } _{s \in {S} _d \setminus {S^{'}} _d}$ represents a assumed full knapsack $\text{Knap}(d)$ defined for the query $q _d$. It reflects the actual schema linking effect of KaSLA and will be used in Eq. (11) to estimate capacity.

• In practice, even an optimized KaSLA might not eliminate all redundant elements. Ideally, the schema linking results would include all matching elements (${S^{'}} _d$ in Eq. (10)) and a small number of high-importance score un-matching elements (${ \lbrace s \mid I _{s,q _d} \geq \min { \lbrace I _{z,q _d}\rbrace } _{z \in {S^{'}} _d} \rbrace } _{s \in {S} _d \setminus {S^{'}} _d}$ in Eq. (10)).
• We use the lowest importance score among the ground truth linking results as a threshold, $\min { \lbrace I _{z,q _d} \rbrace } _{z \in {S^{'}} _d}$. Un-matching elements whose importance score exceeds this threshold are likely to be incorrectly included in the schema linking results by KaSLA.

In summary, Eq. (10) accounts for potential redundant linking in KaSLA in practical application, helping Eq. (11) estimate a realistic and robust capacity.

Empirical Studies:

The detailed experimental results of multiple variants are as follows, we can see that Eq. (10) can help KaSLA achieve optimal performance.

Table W2.2.b Empirical studies of multiple variants of Eq. (10)

W2.3 There are many hyper-parameters in the calculation, e.g., $\tau$,$\alpha$,$\gamma$. Could you explain in detail why each of them is needed? Introducing too many tunable parameters may lead readers to suspect that the experimental performance was achieved through careful hyperparameter tuning, which could indicate poor generalizability.

Thank you for your suggestions regarding hyperparameters. We explained the three hyper-parameters in the KaSLA framework: $\tau$,$\alpha$, and $\gamma$ in detail as follows.

The usages of these hyper-parameters:

• $\alpha$ in the estimation function of importance score: $I({s \mid q,S}) = I _{nomi}{(s \mid q,S)} + \alpha I _{guar}{(s \mid q,S)}$, $\alpha>0$
• $\tau$ in the expectation function of high importance scores: $\mathbb{E} _{I \geq \tau}(S, q) = \mathbb{E}\left[ {\{I _{s,q} \mid I _{s,q} \geq \tau \}} _{s \in S} \right]$, $0<\tau<1$
• $\gamma$ in the estimation function of capacity: $\widehat{C} _q = f _{C}(q) = \gamma \cdot \max \left[ \{ \sum _{s \in \text{Knap}(d)} \widehat{W} _{s,q} \}\right] _{d \in D}$, $\gamma>0$

Explanations:

• We use $\alpha$ to control the magnitude of the contribution from the guaranteed scoring model. A larger $\alpha$ biases the estimation of the importance score towards the guaranteed score, resulting in a more stable and generalized score estimation. Conversely, a smaller $\alpha$ shifts the bias towards the nomination score, achieving a more discriminative score estimation.
• We use $\tau$ to determine whether an importance score is truly important. Elements with an importance score exceeding $\tau$ are considered truly important, and their expectation is used for weight estimation.
• We employ $\gamma$ to adjust the scaling of the estimated capacity. In practical applications, scaling to a larger capacity can include more potential matching elements in the linking results, but it may also overly relax constraints, leading to redundancy. We use $\gamma$ to manage this trade-off.

W2.2 About Eq.(8) and (10), the authors define a little complicated equations to estimate the weight and capacity, respectively. I’m curious if there was any mathematical basis behind the design of these formulas, or you just come up with them all from intuition. If the answer is true, please cite them to make the equations more sound. Otherwise, it would be better to prove that this design is empirical and preferred by comparing multiple variants.

Thank you for your question. We will explain the definitions of Eq. (8) and Eq. (10) in detail and report on related empirical studies. To aid understanding, we'll use table linking as an example, where, in Eq. (8) and Eq. (10), $q$ is a query, $s$ is a table and $S$ is the set of all tables.

Eq. (8): $\widehat{W} _{s,q} = f _{W}(s,q) = \lfloor {(I _{s,q} - \mathbb{E} _{I \geq \tau}(S, q) + 1)}^{-1} \rfloor$ represents the estimation of the weight for table $s$.

Detailed description

There are several important components in Eq. (8):

• $I _{s,q}$ denotes the importance score of table $s$ for query $q$.
• $\mathbb{E} _{I \geq \tau}(S, q) = \mathbb{E}\left[ { {\lbrace I _{s,q} \mid I _{s,q} \geq \tau \rbrace } } _{s \in S} \right]$:
  • ${ {\lbrace I _{s,q} \mid I _{s,q} \geq \tau \rbrace } } _{s \in S}$ represents an important table set, including tables with an importance score greater than $\tau$, where $\tau=0.5$ is the threshold for determining importance.
  • $\mathbb{E} _{I \geq \tau}(S, q)$ is the expected importance score of this important table set.
• $I _{s,q} - \mathbb{E} _{I \geq \tau}(S, q)$ represents the relative importance of $s$. It represents how far the table $s$ is from the important table set. $I _{s,q} - \mathbb{E} _{I \geq \tau}(S, q) > 0$ indicates higher importance, while $I _{s,q} - \mathbb{E} _{I \geq \tau}(S, q) < 0$ indicates lower importance.
• In Eq. (8):
  • The inverse power emphasizes that the weight of $s$ is inversely proportional to its relative importance; the less important it is, the more it is potentially redundant and should have a larger weight.
  • Adding 1 to the denominator prevents division by zero.
  • The use of the floor function simplifies the complexity of the dynamic programming-based knapsack optimization.

Empirical Studies:

The detailed experimental results of multiple variants are as follows, we can see that Eq. (8) can help KaSLA achieve optimal performance.

Table W2.2.a Empirical studies of multiple variants of Eq. (8)

W1.2 This work circumvents this significant issue on the inference delay (including nomination model StarCoder2, guaranteed scorer RoBERTa-Large, extra statistics calculation and dynamic programming). I even did not see one limitation section discussing it. And a quantitive comparison on the inference time w/ and w/o your method is inevitable to trade-off the benefits and disadvantages.

Thank you for your suggestion regarding the trade-off between inference delay and accuracy. We provide the inference time on one 80G A100 GPU for the whole framework, both with and without KaSLA, as follows:

Table W1.2.a Inference time cost of each component in KaSLA

We also provide the SQL generation accuracy for text-to-SQL, both with and without KaSLA, as follows:

Table W1.2.b Comparison of text-to-SQL accuracy with full Schema and KaSLA

As shown in these tables, we can conclude that:

• KaSLA introduces a significant improvement with a marginal inference delay. This delay occurs because we use StarCoder2-15B to generate nomination scores for the full schema, aiming for better semantic understanding, which results in additional time consumption. Reducing the inference time for LLM-based nomination models during full schema processing might be a potential research direction.

W2: Minor issue: some technical design or math is really confusing. Here are some questions:

W2.1 in Eq.(4), you use two scores (both binary $I _{\text{nomi}}$ and float $I _{\text{guar}}$). How about directly use the soft score $I _{\text{guar}}$ as $I _{s,q}$? What is the performance difference? This is important that if a simpler calculation of $I _{s,q}$ can achieve similar expectations, we should always follow the Occam's Razor.

Thank you for this insightful question. We have provided the ablation study results of the importance scores in the table below.

• Only using the soft score $I _{\text{guar}}$ and excluding the binary score $I _{\text{nomi}}$ leads to sub-optimal performance. This is because the RoBERTa-Large based guaranteed model offers only a general representation. This potential lack of obvious differentiation in scoring might introduce confusion, which the StarCoder2-based nomination model will address.
• Only using the binary score $I _{\text{nomi}}$ also results in sub-optimal outcomes. The Starcoder2-based nomination model tends to confirm elements with high confidence, leading to significant differences in scoring. This may result in missing elements that also match, but with medium confidence, which the RoBERTa-Large based guaranteed model should address.

These results highlight the necessity of combining the binary score $I _{\text{nomi}}$ and the soft score $I _{\text{guar}}$ for effective importance score estimation.

Table W2.1.a Ablation study about $I_{\text{nomi}}$ and $I_{\text{guar}}$

Dear reviewer BQ9c,

Thank you very much for your insightful questions and valuable suggestions. We have addressed each point in detail to alleviate your concerns and updated a revision.

W1. Is schema linking task still important for text-to-SQL according to the following concerns？ Since the majority of existing benchmarks have a relatively smaller database schema size, which can mostly afford the prompt length of LLMs. In addition, "The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models" claims that if the schema items can be fully inserted into the prompt, schema linking is not a necessity considering efficiency.

Thanks for your comments. The purpose of schema linking is to increase the accuracy and the performance of the SQL generation. We can present the advantage of the KaSLA schema linking model from the following three points.

The effect of KaSLA

• Accurate schema linking helps LLMs capture useful information and avoid potentially disruptive information. We demonstrated the significant impact of schema linking on SQL generation in the table below.
• Adding KaSLA shows significant improvement over using the full schema. As shown in the table, using KaSLA significantly outperforms using the full schema.

Table W1.a Comparison of full Schema and KaSLA with CodeS-15B as the Text-to-SQL Model

Table W1.b Comparison of full Schema and KaSLA with CodeS-15B as the Text-to-SQL Model

The viewpoint of the provided paper

Thank you for suggesting the paper. We have cited and discussed it in our revision. The paper presents two viewpoints that also support our findings.

• Normal schema linking helps moderately sized models. The paper [1] found that schema linking can significantly enhance SQL generation with moderately sized LLMs, like Llama-3.1-8B (findings are in Section 4.2 of [1]). However, normal schema linking may not improve SQL generation with extremely large LLMs, such as GPT4-turbo, GPT-4o, and Llama 3.1-405b (findings are in Section 4.2 of [1]).
• Accurate schema linking helps extremely large LLMs. The reason schema linking does not improve performance with extremely large LLMs is due to the use of sub-optimal schema linking methods in [1], which limit the LLMs. Since [1] didn’t release their code, we tested our KaSLA on the following SOTA models [2,3,4] with extremely large LLMs and still observed performance improvements.

Table W1.c Integrating KaSLA can further enhance extremely large text-to-SQL models

[1] The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models

[2] E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

[3] Chess: Contextual harnessing for efficient sql synthesis

[4] Text-to-sql empowered by large language models: A benchmark evaluation

For the inference time table, I disagree that the time delay is "marginal" (almost 1.5-2.0 times delay). As you sayed, you use StarCoder2-15B (a relatively powerful and large model) to generate nomination scores (which I also think this relatively large size is needed). But this size of LLM also shows to be time consuming. These overheads maybe unacceptable in tasks requiring timely responses. Thus, please add a separate Limitation section or at least in the Appendix to showcase this extra burden. Each coin has two side and it's OK to leave this trade-off to real scenarios.

Thanks so much for your comments and for further expressing your concerns.

• In light of your suggestions, we have updated our revised PDF. Specifically, on page 10 (from Line 509 to Line 524), we included a separate limitations section just before the conclusion section to discuss our limitation regarding inference delay. As you rightly pointed out, while we strive to deliver more accurate results, minimizing inference delay remains an important task. Your understanding and consideration are deeply appreciated, and we have discussed inference delay in detail in the paper.
• In the future, we hope to reduce the time usage of the nomination model while processing the entire schema while maintaining accuracy. Thanks again for this insightful feedback.

Thanks so much for your response and comments. We sincerely appreciate your recognition of the discussion regarding the various cases of schema linking with different scale LLMs, especially considering diverse downstream applications. It is an honor to engage in this discussion with you.

Dear reviewer BQ9c,

Thanks again for reviewing our work and for your constructive suggestions and comments. To address your latest concerns, we have included a limitations section in the revision to discuss KaSLA's inference delay, and we have provided detailed responses regarding the SQL generation time reduced by KaSLA and discussed the multi-agent group insight about soft/binary score. We hope these responses can address your latest concerns, and thanks for your support.

As the author-reviewer discussion deadline approaches, we would be very grateful if you could share your valuable thoughts with us. If you have any further concerns, we look forward to further discussion with you.

(2) Necessary to propose missing ratio and redundancy ratio

Schema missing ratio: $R _{miss} =\frac{1}{|B|} \sum _{(q,S) \in B} \unicode{x1D7D9}(S _q \nsubseteq \widehat{S} _q)$

Schema redundancy ratio: $R _{redun} = \frac{1}{|B|} \sum _{(q,S) \in B} \underbrace{{(\frac{|\hat{S} _q \setminus S _q|}{|\hat{S}_q|})}^{\sigma}} _{\text{redundancy ratio}}, \underbrace{\sigma = \unicode{x1D7D9}(S _q \subseteq \widehat{S} _q)} _{\text{Non-missing Indicator}}$

Due to the above drawbacks, precision and recall cannot be used in schema linking evaluation. Then we propose the missing ratio and redundancy ratio to evaluate the schema linking accurately.

• Our schema missing ratio can reflect the actual impact of schema linking results to SQL generation by a strict evaluation. It considers an instance a failure if any element is missing, acknowledging success only when no elements are missing. This aligns with the fact that any missing element leads to incorrect SQL generation.
• In Table W4.a, the two examples show a missing ratio of 60% and 100% (lower is better), accurately reflecting the difficulty in generating correct SQL from these schema linking results.
• Lower missing ratio will have higher SQL generation accuracy: The table below shows the column missing ratio and SQL generation accuracy using the same text-to-SQL model with different schema linking models. The results demonstrate that the missing ratio accurately reflects the accuracy of schema linking and directly affects SQL generation. KaSLA achieves the highest SQL generation accuracy with the lowest missing ratio.

Table W4.c Performance on Bird-dev with execution accuracy (EX) for text-to-SQL evaluation and missing ratio for schema linking evaluation. The SQL generation model is a fine-tuned StarCoder2-15B

• Our schema redundancy ratio can provide a meaningful evaluation by calculating redundancy only when all matching elements are present in the linking results. The main difference between precision and our redundancy ratio is that the prediction with element missing will be judged as failure because, in such cases, redundancy is not the sole reason for incorrect SQL generation, thus making its evaluation meaningless.
• In Table W4.b, the two examples show redundancy ratios of 100% and 20% (lower is better), correctly reflecting that the first schema linking result struggles to generate correct SQL while the second one can.
• Lower redundancy and missing ratios will have higher SQL generation accuracy. Using the same experimental setup, the table below shows the column redundancy ratio and corresponding SQL generation accuracy. The results in Tables W4.c and W4.d indicate that only KaSLA achieves the lowest redundancy and missing ratios, resulting in the highest accuracy of SQL generation.

Table W4.d Performance on Bird-dev with execution accuracy (EX) for text-to-SQL evaluation and missing ratio for schema linking evaluation. The SQL generation model is a fine-tuned StarCoder2-15B

Thanks for your valuable suggestions and concerns. We have incorporated all the additional discussions in our revision. We also welcome further discussion on any remaining concerns.

Q3.1: In the paper, when you say schema it is confusing whether you mean table or column or both. It is not until 4.3 that it gets some clarification.

• Since the definitions of knapsack schema linking, metrics, and score and factor estimation are the same for both tables and columns, we did not distinguish them before Section 4.3 to avoid repetition. Such unified definitions aid in the development of KaSLA. Thanks to your suggestion, we have revised the related text for clarity and updated it in the revision.
• We distinguish between tables and columns in Section 4.3, the hierarchical KaSLA strategy, because we treat them differently in a hierarchical process. We first perform table linking, followed by linking columns within the selected tables.

Q3.2 Do you mix schema errors or solve two separate knapsack problems, one for tables and one for columns?

• KaSLA addresses two separate knapsack optimization problems: one for tables and another for columns within the linked tables. After estimating importance scores, KaSLA employs a hierarchical process for schema linking. It starts with table linking using proposed knapsack optimization method, then links columns within the linked tables, guided by the same method.
• This approach is optimal because it reduces the selection space of columns, lowers time costs, and avoids noise and confusion from the columns of unlinked tables. The pseudocode for this process is provided in Appendix C.

Q3.3 The models in the score function, are they trained for tables and columns separately?

• The models in the score function are trained simultaneously for both tables and columns, rather than separately. During the training and inference of the scoring models, addressing both table and column scores simultaneously helps the model integrate semantic understanding and establish relationships across different tables through the attention networks in the language model.
• This simultaneous approach enhances KaSLA's ability to address the challenging multi-table JOIN tasks. The table below shows results from instances involving or not involving the JOIN operation in BIRD-dev. By simultaneously addressing tables and columns, KaSLA outperforms baselines in multi-table JOIN tasks.

Table Q3.3.a Text-to-SQL performance of KaSLA on the BIRD-dev data involving and not involving the JOIN operation

Q3.4 Column selection is conditional, you choose columns from the selected tables; how does that impact the score functions?

• Choosing columns from selected tables is more efficient and accurate than choosing from all tables because it reduces the selection space, lowers time costs, and avoids noise and confusion from unlinked tables' columns. As shown in the table below, the performance of choosing columns from selected tables is better than choosing from all tables in both datasets.

Table Q3.4.a Text-to-SQL performance of KaSLA with choosing columns from selected tables and all tables

Q4: How are the scoring functions trained? Ground truth for which dataset? Dev or train?

• The two scoring functions are trained on the training dataset using its ground truth labels. We only use the dev dataset for evaluations.

W5: The experiments do not include two of the SOTA schema linking works, E-SQL and CHESS, both are on the BIRD leadership. Especially, E-SQL focuses mainly on schema linking and does not use many of the other techniques, like self-consistency, to achieve high accuracy. Experiments should have included at least E-SQL. A. CHESS: Contextual Harnessing for Efficient SQL Synthesis: Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, Amin Saberi B. E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL: Hasan Alp Caferoğlu, Özgür Ulusoy

Thank you for providing these papers. We have included their citations and discussions in our revision. We reproduced E-SQL and CHESS based on the released codes and compared their performance with and without KaSLA.

• Integrating KaSLA can further enhance state-of-the-art (SOTA) text-to-SQL models like E-SQL and CHESS by improving their schema linking. As shown in the table, integrating KaSLA provides a stable improvement for current SOTA models, highlighting its plug-and-play nature and robust schema linking capability.
• Implementation details: We enhanced their schema linking component with KaSLA. CHESS [1] uses a retrieval-augmented chain-of-thought prompting method for schema linking, while E-SQL [2] employs a filtered schema correction strategy. Since CHESS uses GPT-4 and a fine-tuned DeepSeek Coder model, but did not release the latter’s checkpoint, we reproduced each component in CHESS using GPT-4. In line with [2], we used GPT-4o as the base LLM for E-SQL.

Table W5.a Text-to-SQL performance of integrating KaSLA into SOTA baselines (CHESS and E-SQL)

[1] CHESS: Contextual Harnessing for Efficient SQL Synthesis

[2] E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

Q1: Redundancy is a confusing term; do you include the same table/column multiple times in the context? I understand that you do not. This is supposed to capture irrelevant schema elements; i.e. precision. How is this different?

Q2: Similarly, how is missing schemas different than recall?

We have responded to them together with our response to W3; please refer to the above response to W3.

W4: Why introduce new metrics like missing schema and redundancy, while precision and recall of schema linking can fully cover these concepts?

Q1: Redundancy is a confusing term; do you include the same table/column multiple times in the context? I understand that you do not. This is supposed to capture irrelevant schema elements; i.e. precision. How is this different?

Q2: Similarly, how is missing schemas different than recall?

Thanks for noticing this. We first illustrate the limitations of precision and recall in the schema linking task and then explain why we extend them to missing and redundancy metrics for better evaluation.

1) Limitations of precision and recall in Schema

Precision: $R_{precision}=\frac{1}{|B|} \sum_{(q,S) \in B}\frac{|\hat{S}_q \cap S_q|}{|\hat{S}_q|}$

Recall: $R_{recall} = \frac{1}{|B|} \sum_{(q,S) \in B} \frac{|\hat{S}_q \cap S_q|}{|S_q|}$

($B$ is the whole dev dataset, $\hat{S}_q$ is the predicted schema linking results, and $S_q$ is the ground truth.)

They are both general evaluation metrics; however, they have drawbacks that hinder their usage in schema linking.

• The same recall metric values can correspond to different SQL generation accuracies: In schema linking, missing a single table or column (recall < 100%) can significantly decrease text-to-SQL generation accuracy. Averaging recall cannot indicate how many instances are correct. We provide two examples in the table below.
• As shown in the table, in example 1, the average recall is 80%. However, since 3 out of 5 instances have missing elements, only 2 instances can generate correct SQL. In example 2, the average recall is also 80%, but with missing elements in every instance, not a single instance can generate correct SQL. This demonstrates that average recall does not accurately represent actual schema linking results.

Table W4.a Two examples of using recall as a column linking metric for 5 instances—high average recall does not equal high correct linking

• Higher precision metric values may result in lower SQL generation accuracy: Precision is not decisive in schema linking tasks. For instance, if the precision is 100% but the recall is below 100%, there will be 0 correct SQL generation. We provide two examples in the table below.
• As seen in the table, in example 3, all 5 instances have no redundant elements in their linking results, yet all instances still have missing elements. Precision is 100%, but no correct SQL can be generated. In example 4, there are no missing elements, but redundancy exists; therefore, precision is low, but all instances can potentially generate correct SQL. This indicates that precision does not clearly correlate with the success of schema linking results.

Table W4.b Two examples of using precision and recall as column linking metrics for 5 instances—precision cannot reflect linking results accurately

W2: The scoring model is fine-tuning with ground-truth. This requires having a ground truth set, training models, which will only work for benchmark datasets like BIRD and Spider, but is not solution in practice.

Thanks for your comments. With the explicit definition of the knapsack framework, KaSLA demonstrates strong generalizability and transferability.

• KaSLA trained on Spider shows strong performance on BIRD, and vice versa. As shown in the table, even when trained on other datasets, KaSLA provides solid schema linking performance.

Table W2.a Text-to-SQL performance of KaSLA trained on Spider-train but evaluated on Bird-dev