Rethinking Table Instruction Tuning

Thank you for acknowledging that our paper highlights the limitations of existing table LLMs and recognizes TAMA's value to the open-source LLM community.

Regarding your concern

Not sure about the contribution of this paper itself.

We would like to emphasize the three folds of contributions of our work.

First, we systematically examine the overspecialization issues in existing table LLMs, which aligns with the observations by reviewer 8ejo04 in practice. Our analysis reveals the potential trade-offs between task specialization and the model’s general capabilities. We hope our research encourages the community to reconsider whether the performance gains claimed in state-of-the-art table LLMs justify the loss of their general capabilities. We are glad to see that our work has already sparked practice changes in our community – a recent paper [1] has cited our anonymized submission and discusses such trade-offs when they build their table LLM.

In addition, as pointed out by reviewer 8ejo04 and 6ciM, we conduct extensive experiments to demonstrate the influence of hyperparameter settings, which we believe will help practitioners build their customized models efficiently and effectively. Our paper demonstrates that with significantly fewer training samples, we can efficiently fine-tune LLMs and achieve competitive results. It is worth mentioning that our findings align with what reviewer 6ciM observed in their previous experiments in hyperparameter tuning for table fine-tuning, which demonstrates the applicability of our analysis in practice.

Finally, as noted by reviewer 8ejo04, our TAMA model provides a valuable hyperparameter baseline for future table LLM research.

We appreciate your thoughtful feedback and believe that your comments will encourage readers to engage critically with our work, ultimately benefiting our community. In the meantime, we start our paper with the motivation of emphasizing the technical details over telling a compelling story, as these details are crucial as revealed by our experiments and align with the observations in practice as pointed out by reviewers 8ejo04 and 6ciM. We hope our work will assist practitioners in building models more efficiently and effectively, ultimately benefiting the table research community and research for domain-specific instruction tuning in the longer run.

References

[1] Su, Aofeng, et al. "TableGPT2: A Large Multimodal Model with Tabular Data Integration." arXiv preprint arXiv:2411.02059 (2024).

We appreciate your continual engagement in this discussion. Though we respectfully disagree, we believe such conversation will help readers critically evaluate our contributions, ultimately benefiting the community in the longer run.

First, we are glad that we establish the common ground that our systematic exploration has been largely overlooked in the existing literature. And we appreciate that you acknowledge our paper can fill the existing literature gap.

Regarding your concern:

such a practice is more of an engineering effort, which you do not think can justify the acceptance for an ML conference.

While we acknowledge that everyone has their own taste and perspective of research, we respectfully disagree with the characterization that contributions primarily grounded in engineering and empirical exploration are insufficient for ML conferences. Many impactful works published at ML venues advance our understanding through systematic empirical studies or engineering insights that fill important knowledge gaps. For example:

• [1] published at Neurips explores how different data and models affect model capabilities after instruction tuning, offering valuable insights into instruction tuning.

• [2] published at Neurips systematically evaluates popular architectures back then for table tasks, setting up baselines for future research.

These works demonstrate the value of systematic empirical exploration in shaping the direction of follow-up research, and we are deeply inspired by such systematic exploration that these works have carried out.

We would like to emphasize our contributions as follows:

• Examining Overspecialization: We provide a systematic examination of the overspecialization issues in existing table LLMs. We believe this is valuable for the community as it highlights the limitations and potential of current models.

• Systematic Study of Training Setups: We investigate how different training configurations influence model behavior. Our findings show that with proper setups, we can maintain a model’s general capabilities while improving its abilities to deal with table tasks, offering practical guidelines for efficient model development. As you noted, our systematic exploration has been largely overlooked in the existing literature, and our paper can fill the existing literature gap.

• Open-Source Contribution: As you noted, our model is a meaningful addition to the open-source LLM community, providing a strong baseline and facilitating future research in domain-specific instruction tuning.

References

[1] How far can camels go? exploring the state of instruction tuning on open resources.

[2] Revisiting deep learning models for tabular data.

We appreciate your careful and constructive review. We thank you for pointing out that our work identifies the over-specialization issues in existing table LLMs, our experiments are comprehensive, and our research establishes a valuable hyperparameter baseline for future research.

In terms of your concerns,

Closed Code and Weights

We will release our script and model weights upon acceptance. Currently, we did not archive our paper or release any relevant resources online for the sake of integrity in the reviewing process. We want to avoid potential plagiarism or biasing reviewers after making our information public. We hope you can understand our reasoning.

does not examine if the observed trade-offs are specific to table modality or generalizable, such as heterogeneity, permutation invariance, or numeric density.

We have expanded our analysis to assess how table-specific features may influence model performance. To investigate this, we leverage our results of training the Llama 3.1 8B Instruct model for three epochs using 500 examples on each dataset, respectively. We present our results as follows:

We find that the performance degradation is most significant on TabFact. Interestingly, despite TabFact having intermediate numeric density and table-to-question token ratios, it still shows the fastest performance decline.

We hypothesize that this is due to the nature of the task rather than the table-specific features examined. Since FeTaQA and HiTab are table QA tasks, they may possess similar QA form that the model has encountered in its general instruction tuning stage, this may ease the decay of the model’s general capabilities in our fine-tuning stage. However, TabFact is about fact-checking, the input form includes both the table and the claim to be verified, which we suspect may not be as common as the QA data in its general instruction tuning stage. Therefore, the model suffers a more significant performance decay because it needs to update more of its internal knowledge to handle such a task.

further clarified whether observed improvements are due to pre-trained knowledge or specific learning during fine-tuning

We believe that TAMA model has gained ability into handling table data in its fine-tuning stage based on the performance improvement on the out-of-domain test datasets (denoted as S1 and S2 in the paper). Specifically, we choose S1 and S2 as they are synthesized test sets by LLMs from existing literature [1, 2] compared to the training data where all the data is sourced from existing human-annotated benchmarks. The performance on these two out-of-domain datasets is:

While pre-training certainly imparts a foundational understanding of table-related knowledge, our results indicate that table-specific fine-tuning plays a crucial role in further enhancing the model’s capability in handling table data.

[1] Li, Peng, et al. "Table-gpt: Table-tuned gpt for diverse table tasks." arXiv preprint arXiv:2310.09263 (2023).

[2] Wu, Xianjie, et al. "TableBench: A Comprehensive and Complex Benchmark for Table Question Answering." arXiv preprint arXiv:2408.09174 (2024).

Training Size Analysis

We further experiment with various training sizes for each model to observe the impact on performance. Below are the results for Llama 2, QWen 2.5, Mistral v0.3, and Phi 3 8K models at one of the learning rates we select based on our results in the Learning Rate Analysis when we train them for three epochs.

Llama 2 7B Instruct (Learning Rate: 1.0e-6):

QWen 2.5 7B (Learning Rate: 1.0e-6):

Mistral v0.3 7B Instruct (Learning Rate: 5.0e-7):

Phi 3 8K Instruct (7B) (Learning Rate: 5.0e-6):

Key Findings:

Across all models, performance improvement becomes marginal from 600 to 1500 examples, suggesting diminishing returns with larger datasets.

Whether this conclusion still holds true when large models need general capabilities such as reasoning, long text answers, and RAG?

We appreciate your question and would like to share our thoughts here.

In our ongoing works, we have experimented with using a small sample of randomly selected training instances on an NLI explanation dataset. We have observed similar trends: even with only around 10% of the original dataset, Llama 3.1 8B Instruct achieved competitive scores on the test set.

We attribute the phenomenon of training LLMs on limited data still leads to competitive performance to two main factors. First, today’s LLMs are more capable than ever – they can easily pick up patterns, sometimes at the cost of overfitting rather than generalizing broadly. In addition, these models already possess a substantial amount of knowledge and abilities from extensive pre-training, which gives them a head start in fine-tuning.

Second, many tasks share similar reasoning paths. For instance, a math QA dataset may predominantly feature questions about averaging, counting, or addition, and a strong base model may master them with a few examples. This leads to their competitive performance even with limited data.

With these reasons above, we think that for tasks requiring more diverse and complex reasoning paths, or combination of distinct abilities, more training examples may be helpful. The challenge lies in the fact that it is hard to construct a comprehensive dataset – domain-specific tasks often lack diversity, and general benchmarks like RAG or long-text generation require substantial manual labor to craft a comprehensive, diverse and high-quality dataset. Designing high-quality reasoning benchmarks is also difficult, as it requires expert annotators who can think creatively and cover a wide range of reasoning skills.

We have also updated our manuscript accordingly, where the content in blue corresponds to the content we add.

We appreciate your engagement in the discussion and your suggestions, which we believe have made our experiments more comprehensive.

Regarding the paper title

We are currently discussing the title among the authors. One possible revision we are considering is “Rethinking Instruction Tuning: A Case Study on Table Data.” We believe this title better reflect the study's broader scope while emphasizing our experiments on table data. We aim to finalize the title after further internal discussions.

Regarding open-sourcing the script and model weights.

Thank you for pointing us to anonymous.4open.science! Currently, we are categorizing our script, dealing with the license issues for the data, and removing the identity information in our script.

We understand and fully respect your attitude towards our paper and greatly appreciate your engagement in the discussion. Thank you for your constructive feedback, and we hope you have a wonderful Thanksgiving break if you celebrate it :)

We thank you for pointing out that our analysis is comprehensive, our findings are helpful to subsequent research, and TAMA demonstrates an impressive improvement in performance.

In terms of your concerns,

Experiments on other LLMs

We have conducted additional experiments to validate our findings across different models in the full-parameter setup, including Llama 2 7B Instruct, QWen 2.5 7B Instruct, Mistral v0.3 7B Instruct, and Phi 3 small 8K Instruct (7B).

Learning Rate Analysis

We train each model on 500 examples from HiTab, FeTaQA, and TabFact (1,500 examples total) for three epochs. Below are the results for different learning rates:

Llama 2 7B Instruct:

QWen 2.5 7B Instruct:

Mistral v0.3 7B Instruct:

Phi 3 8K Instruct (7B):

Key Findings:

Performance Drop: We observe a significant performance drop happens for every model on the two general benchmarks. Interestingly, for models such as QWen 2.5, when we increase the learning rate from 1.0e-6 to 5.0e-6, it would primarily affect the IFEval dataset rather than MMLU, suggesting that the compromises may happen at different speeds with respect to different aspects of the model's general capability.

Different “Breakdown Point”: The Phi model shows a pronounced performance drop from 1.0e-5 to 5.0e-5, in contrast to Llama, Mistral and QWen models, where the “breakdown point” on the learning rate is slightly smaller, especially for Mistral model, where we see 5 points lose on IFEval from 5.0e-7 to 1.0e-6. Here we list the learning rate we would suggest for practitioners to use if they would fine-tune the following LLMs on table-specific tasks:

Training Size Analysis

We further experiment with various training sizes for each model to observe the impact on performance. Below are the results for Llama 2, QWen 2.5, Mistral v0.3, and Phi 3 8K models at one of the learning rates we select based on our results in the Learning Rate Analysis when we train them for three epochs.

Llama 2 7B Instruct (Learning Rate: 1.0e-6):

QWen 2.5 7B (Learning Rate: 1.0e-6):

Mistral v0.3 7B Instruct (Learning Rate: 5.0e-7):

Phi 3 8K Instruct (7B) (Learning Rate: 5.0e-6):

Key Findings:

Across all models, performance improvement becomes marginal from 600 to 1500 examples, suggesting diminishing returns with larger datasets.