OpenTab: Advancing Large Language Models as Open-domain Table Reasoners

We thank the reviewer for highlighting the novelty of our proposed method, as well as the strong performance registered compared with other baseline models. We respond to reviewer’s questions and comments below.

1. Q: Row selection used, so the method cannot handle complex questions

When humans are writing SQLs to query the large-scale database, one cannot memorize hundreds of thousands of row contents. Instead humans will focus on the table schema and several representative table rows as reference to write the SQL program. We argue that although only a few example rows are selected and fed into LLM for the SQL generation purpose in CODER, it can still mimic human behavior and generate solid SQLs that lead to correct question answering when the tables are large and the question is complex and based on a considerable number of rows. In our response to Reviewer 9EJC (the first reviewer, link), we show the SQL generations of CODER and compare with human labeled SQLs. CODER only reads three selected table rows but still generate high-quality SQLs on the large-scale database of Spider dataset.

2. Q: Minor typos in the manuscript (e.g., by by)

We thank the reviewer for carefully checking the draft. We updated the manuscript for correction.

3. Q: In Figure 3, reader seems to appear after the SQL is selected, but in the explanation, the authors mention that the reader is used to select the SQL. Which is correct?

To clarify, READER is not responsible for the selection of SQL. READER will be input with SQL execution results, SQL program selected as well as context information.

Please let us know if you have further questions or comments.

We thank the reviewer for acknowledging the superiority and novelty of our proposed method and the comprehensiveness of our experiments and ablation studies. Below we answer the reviewer's questions correspondingly.

1. Q: When using BM2.5 for the retriever, are you using the full table?

Yes. Because BM25 is scalable to large tables even in full size, in our experiments BM25 works on full table size. Technically we linearize the table structure into text strings by join table title (if any), header, and cell strings row by row. The BM25 algorithm is built upon the rank_bm25 library [1].

2. Q: For the proposed simple-to-complex strategy for Coder, how is it working with GRSR?

In our implementation, the STC strategy will generate three SQLs and select one for execution and input to READER. GRSR will do ranking based on the selected SQL. We have also tried to concatenate all three SQLs and did the ranking, but this approach didn’t yield better results.

3. Q: For the GRSR, how to deal with SQL queries grounded on multiple gold tables?

Due to the traits of the table QA datasets, in our work we only consider a single-gold-table scenario, where grounded evidence for each question only exists in one single table instead of multiple. One proposal on this is to preprocess retrieved tables to join them into single tables before doing reasoning. We leave further extending to multiple-gold-table settings as future research direction.

4. Q: Are you going to release the code?

Yes we will release our code upon publication of our work.

[1] https://github.com/dorianbrown/rank_bm25

Please let us know if you have further questions or comments.

[1] Borgeaud, Sebastian, et al. "Improving language models by retrieving from trillions of tokens." International conference on machine learning. PMLR, 2022.

[2] Izacard, Gautier, et al. "Few-shot learning with retrieval augmented language models." arXiv preprint arXiv:2208.03299 (2022).

[3] Kweon, Sunjun, et al. "Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table." arXiv preprint arXiv:2305.07288 (2023).

[4] https://github.com/sean0042/Open_WikiTable

[5] Guu, Kelvin, et al. "Retrieval augmented language model pre-training." International conference on machine learning. PMLR, 2020.

[6] Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474.

[7] Liu, Nelson F., et al. "Lost in the middle: How language models use long contexts." arXiv preprint arXiv:2307.03172 (2023).

Please let us know if you have further questions or comments.

4. Q: Didn’t do literature review on RAG papers

Thanks for the suggestion. We drafted a paragraph of discussion about the related works on RAG in Section 6 of the draft. We also put it here for review.

Retrieval Augmented Generators. RAGs [6] leverage retrievers to fetch information from external knowledge database and augment the text input to the language models. REALM [5] and RETRO [1] pretrains the retrieval augmentation framework in the end-to-end manner so that the language model learns to better utilize the retrieved information, while ATLAS [2] jointly trains the retriever as well as the language model. We point out that these well-known RAG models are specialized in the text reasoning domain and cannot be directly applied to table reasoning tasks without fine-tuning, thus inapplicable to the experimental setup of this work.

5. Q: Lost in the middle issue considered?

As in Liu et al. 2023 [7], the “lost in the middle” problem is based on the severe long-context issue of LLM. For OpenTab in the open-domain setting, where the framework needs to inference on multiple retrieved tables, the JR (joint reasoning) strategy has struggling accuracy compared with SR (sequential reasoning) and GRSR (generative reranking & sequential reasoning). Considering the long-context trait of JR, the “lost in the middle” issue could be the cause that is impacting the performance of JR. While for the default strategy GRSR, each time OpenTab does reasoning over one retrieved table and later collectively carries out generative reranking, so that we can effectively evade the “long context” issue and “lost in the middle” problem.

6. Q: From Section 4.2, given that the baseline methods "require full tables to be fed into LLM for reasoning, which may exceed the upper bound of the input token limit, rendering an invalid prediction.", do the results for baseline methods include or exclude these invalid predictions?

For the baseline methods, we include the invalid predictions and treat them as wrong predictions as scalability to large tables is an important property for advanced table reasoners. In experiment most tables in Open-WikiTables and FEVEROUS won’t exceed the 4k token limit of gpt-3.5-turbo and falcon-180b with 2-shot demonstrations. The most severe token overflow issue happens on BINDER, which by default uses 14-shot examples. To fit the long sequence we deploy gpt-3.5-turbo-16k as backbone to run the BINDER.

7. Q: From Table 4, what explains the large gap in difference for BM25 and BM25* for the "w/o Fine-tuning" row? This seems surprising given that BM25 is just supposed to be the authors' replication of BM25* presented in Kweon et al., 2023.

As in the official repo of Kweon et al, 2023 [3,4], there is no public implementation of the BM25 experiments. In the paper the authors did not disclose details of their implementation, either. Due to these reasons we cannot fully reproduce their results. We implemented BM25 on our side following the setting of Kweon et al, 2023 to the most extent and reported the numbers in Table 4. We will open source our code when our work gets published.

8. Q: On the effectiveness of GRSR: how does the effectiveness of GRSR vary with different choices of top-k (where k is number of tables retrieved)?

We refer the reviewer to the third paragraph of Section 5, where we did ablation study to verify the effectiveness of GRSR with varying k. In Figure 5, GRSR improves the QA accuracy with more tables retrieved, while the plain JR and SR methods got more distracted and might get worse accuracy when k increases.

9. Q: In Section 5, explain why using all three SQL difficulties leads to the best performance, as was done in Section 3.2.1

For our STC strategy, SQLs are incrementally generated, and the generation path can also function as reasoning steps, making the generations more likely valid. Moreover the solidness of the generated SQL programs will be verified on the fly, thus we can try to evade invalid SQLs that will either have syntax errors or empty execution results, which is common for a solo SQL generation. To sum up, the STC prompting strategy improves the performance through incremental reasoning as well as flexibility.

We appreciate the reviewer for acknowledging the notable contributions by our work and the novelty of OpenTab to the important Open-domain table reasoning task. Below we address the reviewer's concerns accordingly.

1. Q: The paper claims to provide a new simple-to-complex prompting strategy for text-to-SQL generation using LLMs, …and it is not clear how the LLM is guided to increase the complexity of successive SQL generations

We guided LLM to generate three increasingly complex SQL programs by two means.

• Firstly in the prompt we have such instruction, “Generate 3 SQLite programs with respect to the question separated by [SQLSEP], the complexities of the SQLite programs generated ascend (basic, intermediate, advanced).”. Such instruction works effectively on the instruction-tuned LLMs we studied, which are gpt-3.5-turbo and falcon-180b.
• What’s more we included few-shot examples in the prompt, showing that generated SQLs are in three complexities reflecting different functionalities in order. LLMs learned from the demonstrations well because of their in-context learning ability.

2. Q: …prompt seems intuitively underspecified in guiding the model to generate SQL that focuses on column selection in the first SQL, and column+row selection in the second SQL. In other words, the wording of the prompt doesn't specify what kind of SQL should be generated in each stage. How did the authors verify the SQL was actually of that specific functionality?

As we have explained in the last question, the SQL generation is instructed and guided by the prompt instruction as well as the few-shot demonstrations. Although we didn’t explicitly map [simple, intermediate, complex] to specific functionalities in the instruction, such correlation can be implicitly recovered in the given few-shot demonstrations. Moreover, we designed such demonstrations with the help of gpt-4. We asked gpt-4 to generate SQLs following the [simple, intermediate, complex] types and got back the initial SQL drafts reflecting the different functionalities (row selection, column selection, and beyond), intuitively showing that there could be some implicit alignment between the keywords and the functionalities.

3. Q: Moreover, for this claim of a new strategy (STC) to be sound, one would expect to see it applied to standard text-to-SQL benchmarks like Spider or the new BIRD-SQL.

STC is proposed in the context of end-to-end table QA task, in which SQL programs work as an efficient and scalable interface to query the database and extract relevant information for READER to make inference on. SQL generation is an intermediate tool in the QA pipeline, instead of outcome. While on text-to-SQL benchmarks, ground truth is SQL program itself, which is different from our motivation. While text-to-SQL task is not the focus of this work, we carried out complementary experiments on the “flight_4” database of the Spider dataset to verify the effectiveness of our CODER module with the STC prompting strategy. Below we list the results in the order of question, ground truth, and SQL generated by CODER. We also show the stats of the large-scale database in the table.

How many routes does American Airlines operate?
SELECT count(*) FROM airlines AS T1 JOIN routes AS T2 ON T1.alid = T2.alid WHERE T1.name = 'American Airlines'
SELECT COUNT(*) FROM routes r INNER JOIN airlines a ON r.alid = a.alid WHERE a.name = 'American Airlines';

How many routes end in a Canadian airport?
SELECT count(*) FROM airports AS T1 JOIN routes AS T2 ON T1.apid = T2.dst_apid WHERE country = 'Canada'
SELECT COUNT(*) FROM routes WHERE dst_apid IN (SELECT apid FROM airports WHERE country = 'canada');

What are the names of all airports in Cuba or Argentina?
SELECT name FROM airports WHERE country  =  'Cuba' OR country  =  'Argentina'
SELECT name FROM airports WHERE country LIKE "%cuba%" OR country LIKE "%argentina%";

What are the countries of all airlines whose names start with Orbit?
SELECT country FROM airlines WHERE name LIKE 'Orbit%'
SELECT DISTINCT country FROM airlines WHERE name LIKE "Orbit%";

In how many cities are there airports in the country of Greenland?
SELECT count(DISTINCT city) FROM airports WHERE country = 'Greenland'
SELECT COUNT(DISTINCT city) as airport_cities FROM airports WHERE country LIKE "%greenland%";

We see that our CODER is able to generate solid SQL programs that achieve equivalent functionalities as human labeled, demonstrating the efficacy of our CODER module.