SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging

We sincerely thank the reviewer for their insightful comments and feedback, which have significantly enhanced the quality of our paper.

While SQL-GEN aims to create a unified model that bridges dialect gaps, merging dialect-specific ...

We have included an analysis of the performance of the MoE models on each of the dialect-specific datasets, reported in Table 7. Interestingly, due to the larger capacity of the MoE model compared to the non-MoE expert models, and our proposed method of merging experts while preserving their feedforward layers, the MoE model's performance after fine-tuning did not degrade compared to fine-tuning on a single dialect (As shown in Table 7, we see roughly 3% gain). Compared with models trained on individual dialects (say CodeLlama 7B BigQuery expert in table 7) and a non-MOE model trained on mixture of the three data (CodeLlama 7B “generalist” in table 7) , our unified MOE model outperform they by 4% (31.14 vs 35.44), and 3% (32.83% vs 35.44) respectively. This improvement can partly be attributed to the considerable amount of shared knowledge across the dialect-specific datasets, as all tasks require common features, with only subtle differences stemming from dialect-specific features.

The Mixture of Experts (MoE) approach enables the merging of dialect-specific models, but ...

Thank you for bringing this up, we actually conducted a similar experiment as you suggested, using SFT, which is reported in Table 7 under 'CodeLlama 7B generalist.' Specifically, we combined all the dialect-specific datasets and fine-tuned a CodeLlama 7B model to serve as our baseline all-dialect expert model. We compared this performance with our MoE-based approach, where our proposed method, even without fine-tuning, achieved similar performance to this baseline. After fine-tuning, our MoE model demonstrated an average improvement of 3% in accuracy across all dialects considered in the paper. Additionally, as per your suggestion, we have included a comparison of approximated memory consumption for inference, as reported below, considering our MoE model compared with deploying three different dialect expert models with 7B parameters. As demonstrated in the table below, our MoE model required much less GPU memory compared to keeping three different dialect specific models while keeping on par or better accuracy as reported in Table 7 of paper. Below we will provide an estimated GPU memory requirement for deploying our MoE model comparing to deploying three different expert models across different precisions:

Questions:

How does SQL-GEN handle dialect-specific syntax ...

To support dialect-specific functions, we provide a templated query along with a tutorial on how to use specific keywords or functions. We then ask the model to modify the input template query to incorporate the appropriate dialect-specific functions. Additionally, we have conducted a detailed analysis comparing the number of dialect-specific keywords used in various text-to-SQL baselines. Our approach demonstrates the highest number of samples utilizing dialect-specific keywords, as shown below:

keyword distribution figure: https://anonymous.4open.science/r/SQLGEN-REEBUTTAL-02BD/keyword_distribution_analysis.png

For instance, having the templated query “SELECT * FROM table” and the tutorial on how to use Postgres to_tsquery() function, our SQL-GEN method generated the following new template:

SELECT * FROM table WHERE to_tsvector(column) @@ to_tsquery(literal);

What measures are in place to ensure the synthetic ...

In order to make sure that the generated queries have the high quality we included both semantic and syntactic quality checking by using the existing dialect specific parser to make sure the queries are syntactically correct and also included semantic checks by using a LLM as a judge approach which is called quality checker model in the paper.

We sincerely thank the reviewer for their valuable feedback. In response, in our updated paper, we provided additional explanations for Figures 3 and 4 to ensure the overall workflow presented in these figures is clear and understandable. Furthermore, as suggested, we have updated Figure 5 to enhance its readability, as shown below:

Updated keyword distribution figure: https://anonymous.4open.science/r/SQLGEN-REEBUTTAL-02BD/keyword_distribution_analysis.png

On a side note, it would be interesting to consider cross-lingual transfer methods fo

Thank you for the inspiring suggestions! Cross-lingual Text-to-SQL is indeed an exciting avenue for future exploration with our proposed methodology, and we greatly appreciate your insights. Our MoE model aligns with the philosophy of leveraging cross-dialect transfer between SQL variants to develop a multi-dialect-trained model, which demonstrates superior performance compared to models trained on a single dialect. This represents a step toward achieving effective “cross-lingual transfer methods for Text-to-SQL.” Additionally, training with large-scale, high-quality synthetic data offers significant potential to enhance transfer learning, enabling more data-efficient adaptation to new languages. By using our approach to generate diverse and comprehensive training samples, we aim to prepare models that generalize effectively across languages and SQL dialects, ultimately reducing the reliance on extensive annotated datasets for each new language.

We would like to thank the reviewer for their valuable feedback that has helped us to improve our paper, we really appreciate it.

Novelty We would like to emphasize our novel contributions: we introduced a tutorial-based method for generating synthetic Text-to-SQL datasets across multiple SQL dialects, marking the first approach of its kind and addressing a previously overlooked challenge in the Text-to-SQL domain. Additionally, we proposed a cutting-edge Mixture of Experts (MoE) approach, leveraging dialect-specific initialization to construct a unified model that integrates knowledge from various dialects. This approach not only outperforms dialect-specific models but also streamlines model deployment by consolidating multiple dialect-specific models into a single, efficient model.

The paper presents contributions that, while valuable, appear to have limited ...

Next, we would like to explicitly compare our work with grammar-based approaches, such as the Grappa paper mentioned. Below, we highlight the key distinctions and contributions:

1. Support for Multiple Dialects: Our method is the first in the Text-to-SQL domain capable of generating synthetic samples for multiple SQL dialects in an automated way. It achieves this by leveraging dialect tutorials, enabling the generation of diverse question/SQL pairs that incorporate dialect-specific keywords and functions. In contrast, grammar-based approaches require access to pre-existing question/SQL pairs to induce a context-free grammar, which is often unavailable for many dialects.
2. Flexibility Beyond Templates: Grammar-based methods are inherently limited to specific grammars, restricting their ability to extend beyond predefined templates. Our method, however, generates arbitrarily complex queries, providing greater flexibility. This ability is crucial for avoiding overfitting to canonical query distributions, particularly when generating synthetic data to train LLMs.
3. Automation and Lack of Human Supervision: The Grappa method and similar approaches would necessitate manual annotation to replace entities in seed pairs with corresponding terminal types for each program template. In contrast, our approach is entirely automated, operating end-to-end without any human supervision.
4. Sample Quality Assurance: Grammar-based approaches often populate templates with randomly sampled columns and database values, leading to queries that may lack meaningful context or resemble natural, human-annotated samples. Our method mitigates this issue by incorporating a quality-checking model that filters out low-quality samples, ensuring the generated pairs provide meaningful business insights.
5. Ease of Implementation and Adaptability: Designing a context-free grammar is a non-trivial task, as noted in the Grappa paper, which identifies extending grammars as future work. Our method bypasses this limitation by fusing SQL templates with tutorial documents with LLMs, enabling the generation of sophisticated and contextually relevant queries without the need for predefined grammars. These differences underscore the novelty and practicality of our approach, setting it apart from grammar-based methods and offering significant advantages for generating high-quality, dialect-specific Text-to-SQL data that can be used to build multi-dialect Text-to-SQL systems.

The second weakness is addressed in the next comment ....

improvement of synthetic data is not as good as ...

Although models trained solely on our synthetic queries show slightly lower performance on the BIRD development set compared to those trained on the original BIRD training dataset, it is important to note that the BIRD development set aligns closely with the query distribution of the BIRD training set, inherently favoring models trained on the original BIRD data. More importantly, to measure the value of synthetic data (not overfit on one particular distribution, like BIRD), we evaluated the models on other datasets such as Pagila and the GitHub repository dataset. In these cases, models trained on our synthetic data outperformed those trained on the BIRD benchmark, with a large margin of roughly 20%, highlighting the effectiveness and generalizability of our approach for synthetic data generation across different datasets.

As highlighted in Table 1, when evaluating on the out-of-domain benchmark Pagila, training with the BIRD dataset resulted in a -4.35% reduction in accuracy, whereas training on our dataset led to a 15.22% improvement. A similar trend was observed for the Github_repository dataset, where training on our dataset achieved a 10% improvement, while the BIRD dataset resulted in a -7.5% accuracy reduction.

Furthermore, as demonstrated in Table 6, combining our synthetic examples with the original BIRD dataset improves performance on the BIRD dev set, beyond using the BIRD training set alone by 3-5% . This indicates that our synthetic dataset can enhance model capabilities even when high-quality training data is already available, as demonstrated below: