This paper proposed TAMO (Table as Modality), to inject tabular representations as a separate modality into LLMs to enhance model's robustness towards order permutation and performance on table reasoning tasks. Current table LLMs typically rely on serializing table data while ignoring the relationships between rows and columns and the overall table structure. TAMO uses a hypergraph-enhanced tabular encoder to aggregate structural information by doing average-pooling over cells thus making it robust to order permutations. This structured feature is further injected as a prefix soft prompt, followed by the serialized textual input of the table and the question. The authors also propose StructQA, a new benchmark designed to evaluate table structure understanding and robustness to structural perturbations, such as random row or column shuffling. Results on StructQA and other table QA datasets (HiTab, WikiTQ, etc) demonstrate that TAMO significantly improves performance over existing baselines, achieving a 42.65% average gain. Ablations on different aspects (interpretability, scalability, etc) also provides details for better understanding the proposed approach.

The paper introduces TAMO (Treating Tables as an Independent Modality), a novel framework that treats tabular data as a separate modality integrated with Large Language Models (LLMs). The method embeds a table through the tabular encoder, that imposes a hypergraph structure on cells, and sets of semantically connected cells (rows or columns). The resulting embeddings are then used as soft prompts and can be injected into a wide array of publicly available LLMs. The paper also studies the robustness of other methods on a custom-made dataset.

1. This paper introduces a new dataset, StructQA, which demonstrates that most existing LLMs exhibit poor robustness when textualized tables are used as input.

2. The paper proposes treating tables as a distinct modality, modeling tables using Hypergraph, and introducing TaMo, which employs an encoder to encode Hypergraph aligning with LLMs.

3. Across four mainstream datasets, TaMo achieves significant performance improvements. Further analytical experiments demonstrate that TaMo exhibits stronger robustness compared to using textualized inputs.

The paper proposes TAMO, which treats table representation as an independent modality to enhance large language models (LLMs) reasoning capabilities over tables. TAMO leverages a theoretically permutation-invariant hypergraph to capture the table’s structure. In this hypergraph, nodes represent table cells, and hyperedges represent table headers, capturing the hierarchical structure of tables. By iteratively updating the representation of the nodes and hyperedges at each layer, TAMO learns the table’s structure and integrates this representation into the LLM’s hidden layers.

The authors also introduce a synthetic dataset, StructQA, to evaluate LLM robustness against structural permutations within tables. Experimental results show that TAMO training improves LLM performance on the TableQA task compared to supervised finetuning and demonstrates effectiveness on StructQA.

Dear Reviewers,

We hope this message finds you well. We are deeply grateful for your reviews and recognition, which have significantly strengthened our work. With your insights, we believe TAMO, the first multimodal LLM specifically tailored for the tabular domain, is paving the way for a significant new research frontier.

As the discussion phase ends in less than 8 hours, we’d like to express our thanks individually:

1. Reviewer WgPY: We extend our heartfelt gratitude for your unwavering support and recognition from the very beginning. Your strong rating of 8, combined with the highest level of confidence at 5, means a great deal to us. We are also grateful for your willingness to help convince other reviewers. Your affirmation is a significant encouragement, and we hope your valuable insights will be favorably considered by the ACs and PCs.

2. Reviewer BCkd: We deeply appreciate your meticulous review and detailed suggestions, which have greatly enhanced our work. Your acknowledgment that TAMO "may deserve a note of 8" is encouraging, and we look forward to your latest thoughts, as your feedback is invaluable. We trust that our detailed responses have addressed your concerns.

3. Reviewers aQGU and Vyb4: Thank you for your initial review and your openness to increasing your ratings based on our revisions. We have carefully addressed your concerns and hope the updated paper resolves any lingering doubts. It has been several weeks since we last heard from you, and we would greatly appreciate reconnecting before the discussion phase concludes.

TAMO is opening new doors in multimodal LLMs for table modality, and every reviewer’s contribution will play a critical role in shaping this emerging field. Your support at this stage can make a lasting impact. Let’s shape this frontier together!

Thank you again for your time and valuable input. We look forward to hearing your final thoughts.

Best regards,

The Authors of Submission14078

This paper presents TAMO, the first multimodal LLM for table modality. It also presents StructQA, a benchmark to evaluate the skills of LLMs in understanding structured data. TAMO uses hypergraphs to learn tabular representations which can seamlessly integrate with existing LLMs as soft prompts. Experiments on five datasets show that TAMO yields significant improvements with an average relative gain of 42.65%.

Strengths:

• The research problem of injecting structured information into LLM representations of tabular data is clearly motivated.
• The idea of using hypergraphs to learn structured tabular representations is novel, although this paper can significantly benefit from further discussions on related works using hypergraphs.
• The proposed TAMO dataset can effectively evaluate LLMs in structured tabular data understanding.
• The authors provide comprehensive experiments on five datasets, showing that TAMO enhances structured tabular understanding.

Weaknesses

• There are a few missing experiments, such as evaluation of learned hypergraph representation and generalization to larger tables, which were addressed using the rebuttal period.
• Reviewer BCkd also kindly pointed out many grammatical errors and misuse of certain terms, which were fixed in the revision.

The remaining major issues are:

• Technical writing/presentation (BCkd): “The contributions are occasionally overstated without sufficient evidence”.

• Related to the above point, the current submission lacks discussion and comparison with related works in tabular representations learning (BCkd, aQGU, Vyb4), such as HyTrel (Vyb4). While the authors provide additional discussions in the appendix in the revised version, such critical points should be elaborated and carefully addressed in the main text. The authors shall also consider providing additional experimental results to more directly compare with those related efforts, if possible.

While some reviewers are relatively positive with this submission, I believe this paper would benefit from another round of major revision to clear the concerns regarding related works. Therefore, the decision is reject.

Reject