EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing

Thank you very much for your time and constructive feedback. Below, we will try our best to address the questions raised, hoping to clarify your concerns.

Question #1 on Out-of-domain Dataset

Thank you for suggesting additional experiments on other datasets. We agree that evaluating performance on out-of-domain (OOD) datasets is valuable. Therefore, we conducted experiments on Spider-Dev and Spider-Realistic Dev, two widely adopted text-to-SQL datasets.

• Please note: To evaluate OOD performance, the routers were trained only on the BIRD training set and then directly deployed on Spider-dev without any fine-tuning on Spider-train.

The experiments on Spider-dev and Spider-Realistic dev continue to demonstrate the effectiveness of our routing methods, and the evaluation on this out-of-distribution dataset also shows their generalization capabilities. Most of our routers exhibit improved cost-efficiency. Notably, on Spider-dev, the Qwen DPO and RoBERTa cascading routers achieve performance comparable to consistently using the most advanced methods while reducing token consumption by over 50% and 60%, respectively; on Spider-Realistic dev, Qwen SFT, Qwen Cascading, and Qwen DPO routers reduce token consumption by 46%, 8%, and 55%, respectively, demonstrating impressive cost-efficiency and strong generalizability.

1. Experiments on Spider-dev

2. Experiments on Spider-Realistic dev

Question #2 on "Internal Routing"

Thank you for highlighting this interesting aspect. In DIN-SQL, SQL queries are manually defined and categorized into four levels of complexity: Easy, Non-nested Complex, or Nested Complex. The LLM first classifies the query into one of these categories. Then, it is provided with a few hand-crafted, few-shot examples specific to that category to generate the final SQL. This approach aims to provide suitable examples to assist with different types of SQL generation, rather than primarily focusing on reducing token cost. In fact, DIN-SQL provides numerous examples for all task types (including Easy ones) and does not allocate computing resources based on complexity with the goal of cost reduction.

In contrast, our routing mechanism aims to allocate a more lightweight approach for easier tasks while reserving computationally intensive methods for complex cases, thereby reducing token cost. Furthermore, the routers in our study are trained to learn complexity classification based on the results of schema linking, rather than relying on manually defined classification rules, such as the presence of "nested" queries. This is because the complexity of the text-to-SQL task is multifaceted and not solely related to the presence of nesting. For instance, SQL queries involving computational processes, merging multiple tables, or utilizing functions can significantly increase task complexity. Simple nesting, on its own, may not necessarily pose significant difficulties for advanced models. Therefore, we train routers to learn this classification rather than manually defining the rules.

We will include a discussion of such a comparison in the related work section, thank you for this valuable suggestion.

Question #3 on Latency

Following your suggestions, we tested the time duration on the Bird-dev dataset across different methods (with the same 64 max worker settings). The results are presented in the table below.

Among the base methods, more complex SQL generation approaches yielded better performance but also incurred higher token costs and longer reasoning times to obtain the output SQL. The results also demonstrate the effectiveness of our routers: most routers not only reduced token costs but also decreased latency. Notably, with the Qwen DPO router, we achieved a reduction in token cost of over 40% and a decrease in total time duration of 40%, further highlighting its effectiveness. Moreover, although cascading routers have the advantage in extensibility, their cascading structure can lead to increased processing time, making them less competitive in terms of latency.

Thanks again for your insightful feedback. We hope our response helps clarify and address your concerns. If you have follow-up questions, please feel free to continue the discussion with us.

Dear Reviewer,

Thank you so much for your time and effort in reviewing our paper. Your initial comments are very helpful in refining our paper. We have addressed your comments by (1) supplementary experiments and analysis on two out-of-distribution datasets; (2) detailed discussion with mechanisms in DIN-SQL; (3) report the latency on Bird and obtain new insights.

We sincerely hope that these revisions have addressed your feedback. If you have any further questions, please do not hesitate to discuss them with us.

Best regards,

Authors

Thank you very much for your time and constructive feedback. Below, we will try our best to address the questions raised, hoping to clarify your concerns.

Question #1 on Broader Applications

We appreciate you highlighting the potential for our approach to be applied to broader reasoning tasks. It is true that our method could potentially be adopted for other general reasoning tasks, such as math reasoning, to reduce computational costs.

However, not all reasoning tasks are as token-consuming as text-to-SQL. Text-to-SQL is special because of its close relationship with databases: a large database can have a very long and complicated schema, making it extremely token-consuming and challenging for LLMs to analyze and generate correct SQL queries. It can cost tens of thousands of tokens or even more for LLM to just deal with a single query in the Bird dataset, and often, multiple times of token investment may yield only marginal improvements (~1%) on leaderboards.

Also considering the critical role of text-to-SQL in democratizing database access for novice users, we think there is an urgent need for more sustainable text-to-SQL approaches to enhance its practicality, which motivates us to explore and focus on this special and important field.

Question #2 on Experiments for More Datasets

Thank you for suggesting additional experiments on other datasets. We agree that evaluating performance on out-of-domain (OOD) datasets is valuable. Therefore, we conducted experiments on Spider-Dev and Spider-Realistic Dev, two widely adopted text-to-SQL datasets.

• Please note: To evaluate OOD performance, the routers were trained only on the BIRD training set and then directly deployed on Spider-dev without any fine-tuning on Spider-train.

The experiments on Spider-dev and Spider-Realistic dev continue to demonstrate the effectiveness of our routing methods, and the evaluation on this out-of-distribution dataset also shows their generalization capabilities. Most of our routers exhibit improved cost-efficiency. Notably, on Spider-dev, the Qwen DPO and RoBERTa cascading routers achieve performance comparable to consistently using the most advanced methods while reducing token consumption by over 50% and 60%, respectively; on Spider-Realistic dev, Qwen SFT, Qwen Cascading, and Qwen DPO routers reduce token consumption by 46%, 8%, and 55%, respectively, demonstrating impressive cost-efficiency and strong generalizability.

1. Experiments on Spider-dev

2. Experiments on Spider-Realistic dev

Question #3 on Various Router Implementations

We actually include various router implementations in our experiments: beyond Qwen2.5-0.5B (a representative small casual LLM) with supervised fine-tuning, cascading, and direct preference optimization, we also include investigations on RoBERTa-base (a representative encoder-only model, showcasing a different model architecture) as well. Moreover, we also use KNN as a straightforward heuristic approach for comparison.

Question #4 on Larger LLMs as Backbone

To balance experimental expenses, we deploy the more affordable GPT-4o-mini as the backbone model for a fair evaluation. Following your suggestions, to investigate a larger LLM as the backbone for a comparison, we test with a larger GPT-4o. Considering the significant cost of using GPT-4o as the backbone in our task, we test with three of our representative routers due to time and funding constraints.

As shown in the table below, the tested routers are also proven effective when we deploy a larger LLM as a backbone model (GPT-4o here). RoBERTa Cascading router and Qwen DPO router reduce token consumption by 41% and 44%, respectively, without sacrificing performance, significantly improving the TEP metric. This also suggests good model transferability of our framework.

As demonstrated in the table below, the tested routers also proved effective when using a larger LLM (GPT-4o here) as the backbone. Specifically, the RoBERTa Cascading router and the Qwen DPO router reduced token consumption by 41% and 44%, respectively, without sacrificing any performance, thereby significantly improving the TEP metric. This result also suggests good model transferability for our framework.

Thanks again for your insightful feedback. We hope our response helps clarify and address your concerns. If you have follow-up questions, please feel free to continue the discussion with us.

Thanks to the authors for their clarification. I am happy to raise my scores.

Thank you very much for your time and constructive feedback. As we mentioned in the limitation section, labeling training data indeed introduces an upfront cost (although this expense becomes increasingly amortized with wide adoption).

For scenarios with limited training data, we think our cascading routers offer viable options. This is because for cascading structured routers, integrating a new method only necessitates the addition of a corresponding binary router, without requiring retraining of the original router. Consequently, this approach facilitates easier extension and integration while simultaneously reducing the demand for extensive training data. Therefore, we agree that pursuing more scalable and generalizable routing for text-to-SQL is an important topic for future studies.

We appreciate your insightful feedback once again.

Thank you very much for your time and constructive feedback. Below, we will try our best to address the questions raised, hoping to clarify your concerns.

Question #1 on Schema Linking

1. Perfect schema linking is indeed challenging, its primary goal is to identify relevant tables and columns for the current task. As long as we maintain a high recall, some noise is tolerable. This is because missing a column will negatively impact SQL generation, whereas a redundant column can often be ignored during the SQL generation stage. This makes redundant noise more acceptable. In this regard, recall is prioritized. We examined the schema linking performance of GPT-4o-mini in our experiments and found that it achieves a high recall of 93% while reducing the number of columns by over 90% on average. This suggests that the schema linking in our experiments is sufficiently robust.
2. Moreover, as schema linking is a standard procedure in text-to-SQL, we are motivated to utilize the results, which is why we position the routers after schema linking rather than before it. The database schema in the Bird dataset can be quite large, making it very difficult to route tasks solely based on the query and the full database schema. In contrast, since schema linking is a standard procedure, its intermediate results are valuable. Although these results may contain some noise from redundant elements, we still filter out over 90% of columns on average. This provides much more informative references for the routers compared to the full database schema. Therefore, we believe that placing routers right after schema linking is an appropriate design choice to utilize the linked schema.

Question #2 on Routing

Thank you for highlighting the relevant work of RouteLLM; we have cited it in the related work section. RouteLLM is an illuminating work on LLM routing and inspired some of our router implementations.

RouteLLM proposes supervised fine-tuning (SFT) open-source LLMs to perform binary selection of the more suitable LLM using a special classification token. Our classification-based routers trained through SFT are similar to this approach. However, our method differs in that it supports multi-class classification by adding a linear classifier head. Furthermore, we introduced Direct Preference Optimization (DPO) to further help routers learn preferences, and these DPO-trained routers demonstrate better routing performance compared with SFT-trained ones. They are also the most effective routers in our studies. We believe that investigating and integrating more advanced routers could be an interesting topic for future research.

Thanks again for your insightful feedback. We hope our response helps clarify and address your concerns. If you have follow-up questions, please feel free to continue the discussion with us.

Dear Reviewer,

Thank you so much for your time and effort in reviewing our paper. Your initial comments are very helpful in refining our paper. We evaluate the schema linking performance in our experiments, which is robust. We also clarify the position of routers, as well as the connection and the difference between RouteLLM and our SFT-based routers.

Additionally, we conducted further experiments during the discussion period, which are summarized in the general response. We kindly invite you to refer to those results if you are interested in.

We sincerely hope that these revisions have addressed your feedback. If you have any further questions, please do not hesitate to discuss them with us.

Best regards,

Authors