PoTable: Programming Standardly on Table-based Reasoning Like a Human Analyst

W5. It is suggested to discuss why "removing data type cleaning (w/o Dty. Cle.)" gets the lowest scores.

Thank you for your suggestion. Among all ablated settings, w/o Dty. Cle. always reaches the lowest accuracy. This is because, in our predefined code, all cell contents are stored as string types. In a no-stage-split setting (only Reason), the code generation module can easily consider the need for data type cleaning for this part given the initial code base of table variable definition. However, in w/o Dty. Cle. setting, such cleaning can only be performed after the row selection stage, which leads to more content in the code base so that the planner and code generator may find it easier to ignore such details of type cleaning. Therefore, the results indicate that data type cleaning is a very important stage, and the framework may ignore details with unreasonable top-level stage split. We will revise the corresponding analysis in the new revised manuscript.

W6. The max tokens of the backbone models used in the study is 2048, with possible risks of exceed input.

In fact, the max_tokens restricts the output token number of LLM generations instead of the input one. Our selected base LLMs accept 128K input tokens, so there is no risk of exceeding input.

W7. Would suggested operations be contradicted by the claim of "open-world space without any constraints"?

In fact, the suggested operations in prompts were not contradicted by the claim of "open-world space without any constraints". During the reasoning stage, PoTable may plan and execute some operations that are not recommended, such as extracting some intermediate variables, as shown in the case study. Although we will no longer use the expression "open-world space" in the revised manuscript, we still want to point out that the relevant hints are only to make the planner aware of what should be done in the reasoning stage. In this way, PoTable integrates logical top-level guidance with adequate autonomy in tool-based execution.

Thank you for your constructive feedback. We have carefully considered your suggestions and revised our manuscript. Here are our detailed responses to your concerns.

W1. I don't think Initialization and Final Answering are important enough to be count as stages.

We understand why you have raised this concern. In our evaluated tasks, the initialization stage used pre-defined code to store a standardized single table, while the final answering stage only needed to output the final reasoning answer. In fact, in some complex scenarios, these two stages require some more refined adjustments. For example, when storing hierarchical or multiple tables, we may require more complex planning and code generation in the initialization stage. The processing of multi-hop question answering and multiple key variable combinations follows the same principle. In order to further expand in more complex scenarios in the future, we define these two parts as distinct stages.

W2. The claim that your analysis stage split is human-like is too arbitrary. If is actually does, please cite evidence in the paper.

We would apologize for our previous expression that have impacted your understanding of our study. This prompted us to rethink and refine the our motivation and contributions. Specifically, we overemphasized that our stage split was standard rather than conveying more essential concept of integrating logical top-level guidance. In this way, PoTable enjoys two advantages that make it a distinguished data analyst. (1) Accuracy: PoTable can easily plan coherent operation chains under precise and specific sub-goals with less possibility of misleading or missing steps, producing more accurate results. (2) Explainability: PoTable follows suitable structured top-level guidance with full operation execution, making it easier to verify the completeness and accuracy of the reasoning process. Furthermore, the logical stage split was inspired by human data analysis and mining processes [1][2], aiming to make the reasoning results as accurate and explainable as possible. In the revised version, we will describe the stage split as "relatively standard" because it indeed represents a widely-accepted top-level guidance among most researchers. We sincerely thank you for your suggestions, which have significantly improved the motivation and contributions of our work.

W3. The evaluation datasets are not extensive enough.

Some previous studies (e.g., Dater, Chain-of-Thought) evaluate their method through three commonly-used datasets, i.e., WikiTQ, TabFact and FeTaQA since they all adopt LLM-query as the main reasoning mode. However, PoTable adopts full program execution to generate answers, being not able to handle free-form answer generation in FeTaQA directly. Therefore, we only adopt WikiTQ and TabFact as the evaluation datasets. Following Dater, we also evaluate the dev set of WikiTQ, and the fine-grained ablation study results in fact reveal that the dev set follow somewhat shifted distribution towards the test set.

W4. The results analysis in the seconde paragraph in Sec 3.2 could be further improved.

Thank you for your suggestion. The detailed analysis of Binder and Chain-of-Table has already presented in the current manuscript. As for Dater and TabSQLify, we indeed observe there unstable accuracy difference across different LLM bases and datasets. Based on their implementation code and results, we can only make assumptions that their module and prompt design lack robustness under different LLM bases, while PoTable demonstrates no such disadvantage. We will revise the corresponding analysis in the new revised manuscript.

[1] Fayyad U, Piatetsky-Shapiro G, Smyth P. The KDD process for extracting useful knowledge from volumes of data[J]. Communications of the ACM, 1996, 39(11): 27-34.

[2] Mariscal G, Marban O, Fernandez C. A survey of data mining and knowledge discovery process models and methodologies[J]. The Knowledge Engineering Review, 2010, 25(2): 137-166.

Thank you for your constructive feedback. We have carefully considered your suggestions and revised our manuscript. Here are our detailed responses to your concerns.

W1. Unclear unique contribution of the proposed method of non-standard logical splits and constrained operation pools. They need revision.

We would apologize for our previous expression that have impacted your understanding of our study. This prompted us to rethink and refine the our motivation and contributions. Specifically, we fail to convey more essential concept of integrating logical top-level guidance in the first version of the manuscript. In this way, PoTable enjoys two advantages that make it a distinguished data analyst. (1) Accuracy: PoTable can easily plan coherent operation chains under precise and specific sub-goals with less possibility of misleading or missing steps, producing more accurate results. (2) Explainability: PoTable follows suitable structured top-level guidance with full operation execution, making it easier to verify the completeness and accuracy of the reasoning process.

We sincerely thank you for your suggestions, which have significantly improved the motivation and contributions of our work.

W2. It is unclear why the stage split is standard, and it is unknown how this split can be generally applied to all table-reasoning tasks.

As you concerned, we overemphasized that our stage split was standard in the first version of the manuscript. Our vague expression made you mistakenly believe that our division was the most standard, which was contrary to what we actually want to express. On the other hand, the logical stage split was in fact inspired by human data analysis and mining processes [1][2], aiming to make the reasoning results as accurate and explainable as possible. Therefore, in the revised version, we will describe the stage split as "relatively standard" because it indeed represents a widely-accepted top-level guidance among most researchers.

As for general applications, our PoTable is actually extensive and scalable in some scenarios, yet needs a little customization in specific scenarios during implementation. This is because that all these stages are class instances of PoTableBlock in our implementation code. Specifically, for scenarios of multi-table and hierarchical tables, we may design a more sophisticated instructions for the initialization stage to store multiple tabel data frames (e.g., store them into df1, df2 is a naive method) and tables with multi-level indexes (as work [3] designed as a distinct pre-processing stage), maybe with a more fine-grained stage split. As stated in the conclusion, we hope to enhance our approach in the future to better accommodate more advanced task scenarios.

Thank you again for your suggestion, which inspires us to revise our manuscript in a more comprehensive manner.

[1] Fayyad U, Piatetsky-Shapiro G, Smyth P. The KDD process for extracting useful knowledge from volumes of data[J]. Communications of the ACM, 1996, 39(11): 27-34.

[2] Mariscal G, Marban O, Fernandez C. A survey of data mining and knowledge discovery process models and methodologies[J]. The Knowledge Engineering Review, 2010, 25(2): 137-166.

Thank you for your constructive feedback. We have carefully considered your suggestions and revised our manuscript. Here are our detailed responses to your concerns.

W1. This work lacks novelty, since the highlighted human-like reasoning stage design lacks justification.

We would apologize for our previous expression that have impacted your understanding of our study. This prompted us to rethink and refine the our motivation and contributions. Specifically, we overemphasized that our stage split was standard rather than conveying more essential concept of integrating logical top-level guidance. In this way, PoTable enjoys two advantages that make it a distinguished data analyst. (1) Accuracy: PoTable can easily plan coherent operation chains under precise and specific sub-goals with less possibility of misleading or missing steps, producing more accurate results. (2) Explainability: PoTable follows suitable structured top-level guidance with full operation execution, making it easier to verify the completeness and accuracy of the reasoning process.

Furthermore, the logical stage split was inspired by human data analysis and mining processes [1][2], aiming to make the reasoning results as accurate and explainable as possible. In the revised version, we will describe the stage split as "relatively standard" because it indeed represents a widely-accepted top-level guidance among most researchers. We sincerely thank you for your suggestions, which have significantly improved the motivation and contributions of our work.

W2. Pipeline design choices not well-motivated or -justified, since it is unknown why only use few-shot examples at the final stage.

Our framework leveraged two capabilities of the LLM: operation planning and code generation. In the planning phase, we use few-shot examples across all stages since they have clear and precise sub-goals. As for code generation, however, we need to generate appropriate code for each operation. During this process, we don't know the specific content of each fine-grained operation in advance, but only know that it is a crucial component to accomplish the stage goal. Therefore, we cannot provide any examples for few-shot learning. The final stage's output for the answers is consistent across all samples in the specific task, and we must ensure that this stage can generate executable code with the print() method to produce valid outputs. Hence, we use few-shot examples for this stage.

In other words, your mention of only use few-shot examples at the final stage means "during code generation, we only use few-shot examples at the final stage", since in the planning phase we always use few-shot examples for better planning.

W3. This method is somewhat overfitted to a certain type of table reasoning, leading to poor generalization over other types of questions (e.g., complex queries, merging, multi-step).

Our PoTable is actually extensive and scalable, yet needs a little customization in specific scenarios during implementation. This is because that all these stages are class instances of PoTableBlock in our implementation code. Specifically, for scenarios of multi-table and hierarchical tables, we may design a more sophisticated instructions for the initialization stage to store multiple tabel data frames (e.g., store them into df1, df2 is a naive method) and tables with multi-level indexes (as work [3] designed as a distinct pre-processing stage), maybe with a more fine-grained stage split. Regarding the scenarios you mentioned (e.g., complex queries, merging, multi-step processes), our current method requires further optimization to address these more intricate task scenarios. The core contribution of this paper does not focus on these high-level tasks. As stated in the conclusion, we hope to enhance our approach in the future to better accommodate more advanced task scenarios. Thank you again for your suggestion, which inspires us to revise our manuscript in a more comprehensive manner.

[1] Fayyad U, Piatetsky-Shapiro G, Smyth P. The KDD process for extracting useful knowledge from volumes of data[J]. Communications of the ACM, 1996, 39(11): 27-34.

[2] Mariscal G, Marban O, Fernandez C. A survey of data mining and knowledge discovery process models and methodologies[J]. The Knowledge Engineering Review, 2010, 25(2): 137-166.

[3] Cao Y, Chen S, Liu R, et al. API-Assisted Code Generation for Question Answering on Varied Table Structures[C]. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 14536-14548.

W3. Programming-based table agents may struggle with three key scenarios: multi-table reasoning, text-table hybrid reasoning and irregular structured table reasoning (e.g., hierarchical tables).

This question is essentially the same as Q1, namely:

Q1. How do you envision PoTable could be extended to handle multi-table scenarios or hybrid reasoning with both tabular and textual data or irregular table structures?

Our PoTable is actually extensive and scalable, yet needs a little customization in specific scenarios. This is because that all these stages are class instances of PoTableBlock in our implementation code. Specifically, for multi-table and hierarchical tables, we may design a more sophisticated instructions for the initialization stage to store multiple tabel data frames (e.g., store them into df1, df2 is a naive method) and tables with multi-level indexes (as work [1] designed as a distinct pre-processing stage), maybe with a more fine-grained stage split. In text-table hybrid reasoning, as our reasoning results are generated by full program execution, the current PoTable cannot handle it directly. Therefore, another final LLM-query step is required as an augmentation to summarize its comprehension and the executed result to draw the final conclusion.

Q2. As LLM capabilities continue to advance, would you agree or maybe disagree that the benefits of structured workflows might need to be reconsidered?

With the improvement of capable LLMs, numerous studies no longer focus on structured design but autonomous exploration to stimulate the potential LLMs. Despite their success, the completely autonomous exploration method is prone to losing some details or planning some redundant operations in some complex situation reasoning scenarios, and it is difficult to obtain reliable results in some specific field scenarios because it is time-consuming for verification. Consequently, we hope to inject logical top-level guidance in a suitable manner into our reasoning framework to achieve a balance between structure and autonomy. The experimental results also prove the effectiveness of this idea (as can be seen from the comparison in Table 1). In the future, in more complex and special scenarios, we hope to continue to explore more effective balance methods so that the correctness and explainability of the reasoning results can be well guaranteed. We would like to thank you again for raising this issue, which promotes us to optimize the expression of our core motivation and contribution of this study.

[1] Cao Y, Chen S, Liu R, et al. API-Assisted Code Generation for Question Answering on Varied Table Structures[C]. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 14536-14548.

Thank you for your constructive feedback. We have carefully considered your suggestions and revised our manuscript. Here are our detailed responses to your concerns.

W1. The idea of using LLMs with pandas for table reasoning has already been previously explored in various works. The main contribution will be in providing a more refined (and maybe fixed) workflow for table reasoning, which might have diminishing returns with more capable models.

As you mentioned, using LLMs with pandas is not the core innovation of this study. Instead, inspired by the stage-split principles that a distinguished tabular data analyst may follow, our study inject logical top-level guidance into the tool-based reasoning framework by spliting several distinct logical stages. In this way, our PoTable can easily plan coherent operation chains under precise and specific sub-goals, and the completeness and accuracy of the whole reasoning process can be easily validated.

Although the improvement may decline as the capabilities of the base LLM increase, there is a rising demand of numerous domain-specific reasoning scenarios rather than general reasoning, which might need more specific intrinsic guidance or knowledge. Compared with feeding LLMs with massive data from different domains, similar top-level guiding thinking may greatly improve the reasoning effect in a faster and more economical way. This is also a major direction we will explore in the future. Thanks again for your inspirations to us.

W2. The paper lacks a crucial analysis of token efficiency and alternative approaches.

Many thanks for your valuable suggestion, which promotes us to conduct efficiency experiments on TabFact (S) for GPT-based PoTable and other three representative baselines. Please check the results in Table 1.

Table 1. Efficiency experiments on TabFact (S) for GPT-based methods. For Binder, Dater and Chain-of-Table, we compared the results of single LLM generation (n=1, explained in Appendix A) and default LLM generation (Default, which means n_samples follows their claimed settings in the corresponding article). In addition, we estimate the LLM generation counts of each method under default setting.

Based on the result in Table 1, we make the following analysis.

• We conducted experiments of baselines under Default settings, and noticed that they did achieve some improvement compared to results of n=1 settings. However, PoTable always follows the n=1 setting yet outperforms these baselines, which fully demonstrates the effectiveness of PoTable.

• On the other hand, we compared the LLM generation counts to evaluate the efficiency of these methods. In Binder and Dater, the generation counts are fixed as 50 and 100 respectively, while the ones of Chain-of-Table and our PoTable are fluctuated dynamically. Therefore, we calculated the empirical average counts of each module and rounded them up as their estimation (shown in Details). It can be seen that PoTable has much fewer LLM generation counts compared to previous baselines.

• In addition, the LLM generation count increase of PoTable than PoTable (only Reason) is limited, while the accuracy improvement is more than 3%. Among them, the logical stage split slightly increases the LLM generation counts of Operation Planning and Code Generation, but under clearer stage guidance, the planning module is less likely to produce wrong plans or overlook important details. Such results validate the efficiency and effectiveness of our PoTable.

In summary, the effectiveness of our method comes from the design of logical stage planning and code execution rather than multiple LLM generation. This part of the analysis results will be updated to the next submission as the optimization part of our work. Thank you again for your valuable suggestion that inspires our study to be better improved.