Table as a Modality for Large Language Models

We thank the reviewer for their feedback and for finding our hypergraph-based approach interesting.

1. W1 / Q1 (Weakness of "Inference only" baseline):

The reviewer is correct: "Inference only" is a zero-shot baseline, chosen deliberately. First, there is no universally adopted few-shot methodology for the TableQA domain. Unlike other NLP tasks, each sample consists of a unique table and question. Constructing a few-shot prompt would require including multiple, often large, tables as examples, which is frequently impractical due to the context window limitations of most models.

Second, the zero-shot baseline probes out-of-the-box LLMs’ intrinsic structural understanding, especially state-of-the-art table reasoners like TableLlama. Our results confirmed that even such models struggling here reveals a fundamental limitation, making this baseline a key diagnostic, not just a performance floor.

Finally, our core improvement claims rely on comparisons with stronger trained baselines (Prompt Tuning, LoRA, SFT), where TAMO shows significant advantages (Table 2). We will label "Inference only" as "zero-shot" in revisions for clarity.

2. W2 / Q2 (Clarification of "Specialist SOTA" and evaluation on StructQA): This is an excellent point. The term is indeed vague. In our revision, we will update the caption of Table 2 to explicitly name the specific model and provide its citation for each dataset. The specific models are:

• For HiTab, the Specialist SOTA is TableLlama, which is based on Llama2-7B and underwent extensive supervised fine-tuning (SFT) on the HiTab training set along with other table QA datasets. The score reported in the original paper is 64.71, which we reproduced as 63.76.

• For WikiTQ and FetaQA, the SOTA is CABINET, which improves LLM performance by using an external scorer to assign higher weights to question-relevant table cells, thereby filtering out noisy ones.

• For WikiSQL, the SOTA is SeaD, which enhances text-to-SQL capabilities through a schema-aware denoising objective.

It is important to note that our purpose in including these specialist SOTAs is to highlight different methodological directions. Unlike our TAMO framework, which directly enhances the LLM's end-to-end understanding of the table as a modality, methods like CABINET and SeaD rely on external modules or task-specific pre-processing.

These models were not evaluated on our new StructQA benchmark because they often involve complex, dataset-specific pipelines that are not readily adaptable to a new task format without significant engineering effort. Our goal with StructQA was to test the core structural understanding of more generalist LLM-based approaches.

3. W3 (Generality of the Approach):

This is a fair point about the term "general." We use it in two senses:

1. Generalization across structural variations, such as the row and column permutations tested in StructQA, where TAMO shows superior robustness (Figure 6) .

2. Architectural generality, as TAMO is a "plug-and-play" module that can be integrated with various mainstream LLMs (e.g., Llama2, Mistral, Llama 3) without intrusive architectural modifications . This contrasts with methods that require re-training an LLM's internal position encodings.

We will refine our language in the paper to be more precise about these two aspects of generality.

Thank you for the clarification. This is an excellent question that gets to the heart of our paper's contribution, and we appreciate the opportunity to highlight the key results that directly address it.

Let us first briefly recap the core premise of our work. Our investigation began with a foundational finding: serializing tables into text, while convenient, causes LLMs to lose a precise understanding of the table's structure. This was validated in our StructQA benchmark, where even advanced models struggled with simple structural tasks like permutation. To address this information loss, we proposed modeling the table's structure explicitly using a graph representation and treating it as a distinct modality.

You asked if a small amount of fine-tuning would have been sufficient to improve performance, and whether our graph representation was truly necessary. This is precisely the central question our main experimental setup in our paper's Table 2 was designed to answer. In Table 2, for each of the three standard fine-tuning paradigms (Prompt Tuning, LoRA, and full Supervised Fine-tuning), we directly compared two settings: a text-only baseline (representing standard fine-tuning) versus our TAMO framework (which adds the table modality by graph).

The results, summarized below, unequivocally show that while standard fine-tuning does improve performance, it is not sufficient. Across all datasets and all tuning methods, adding our TAMO modality provides a substantial additional performance gain.

This direct, controlled comparison proves that the performance gains are not just from fine-tuning, but specifically from the structural information captured by our graph representation. Therefore, to answer your question directly: yes, the graph representation and treatment of the table as a modality are indeed necessary to overcome the fundamental limitations of text serialization, a fact that standard fine-tuning alone cannot resolve.

Furthermore, to demonstrate the broad applicability of our findings, we have validated this conclusion across multiple LLM families. Our main paper shows these results on Llama2-7B and LLaMA 3.1 8B. We have since conducted additional experiments on the newer Qwen2.5-7B-Instruct and Qwen3-8B models, with the same consistent outcome:

In summary, the evidence from multiple fine-tuning methods and across diverse LLM families robustly confirms that our TAMO framework is a necessary and highly effective solution.

We hope this detailed explanation, grounded in the core motivation and experimental design of our paper, fully addresses your concerns. Thank you again for your time and engagement.

We thank the reviewer for their detailed feedback and for recognizing the value of our StructQA benchmark.

1. W1, W3 (Novelty and Insufficient Baselines):

We thank the reviewer for the reference to TabGLM. While prior works have combined graphs and text, TAMO's novelty lies in pioneering the treatment of table structure as a distinct modality within the modern large language model instruction-tuning paradigm. Our framework is designed as a general, plug-and-play module for various LLMs, which is a significant advancement. We will revise our introduction and related work to discuss TabGLM and further sharpen our positioning.

Regarding baselines like TAPAS/TAPEX, we have run these experiments on our StructQA benchmark. The results are as follows:

The results clearly show that TAMO dramatically outperforms these baselines on the core structural understanding tasks (StructQA and Permutation Accuracy). It is crucial to consider the architectural differences: TAPAS is a BERT-based encoder, and TAPEX is a BART-based encoder-decoder model. In contrast, our TAMO framework is specifically designed as a multimodal extension for decoder-only LLMs, which represent the current dominant architecture. While TAPAS and TAPEX show higher robustness, this is largely an artifact of their very low base accuracy; it is easier to remain consistent when a model is consistently incorrect. TAMO's performance demonstrates a far superior grasp of table structure. We will add a new table and this analysis to the appendix in our revised manuscript.

2. W2 (Claim of "Cost-Effective"):

We appreciate the request for clarification. The "cost-effective" claim is relative to the cost of fine-tuning the LLM itself. As shown in our efficiency analysis in Appendix C.2 (Figure 7), TAMO's training time (in the frozen LLM setting) is significantly less than LoRA or full supervised fine-tuning (SFT) because we only train the small encoder and projector. Furthermore, to clarify the architecture in Figure 4, while there are two input streams, our framework only introduces one new encoder (the table encoder). The textual input is processed directly by the native embedding layer of the decoder-only LLM, which does not require a separate text encoder. This is fundamentally different from models like TAPEX or TaBERT, which employ dual-encoder architectures and are thus computationally more demanding. We will clarify this context in the main text.

3. Q1 (Why only WikiTQ for StructQA?):

We chose tables from WikiTQ because it offers a large and diverse collection of high-quality, real-world tables from Wikipedia. These tables are structurally simple for humans to understand. The fact that modern LLMs still struggle with these seemingly simple structures makes their failure more pronounced and powerfully demonstrates the core problem we aim to solve. This diversity and simplicity were essential for creating a benchmark that robustly tests a model's fundamental structural understanding. We will add this justification to Appendix A.

4. Q2 (Dependence on Serialization Template):

This is an insightful point. While we used a consistent format for fair comparison, our core hypothesis is that any serialization format inevitably leads to a loss of structural information. TAMO is designed to compensate for this fundamental loss, regardless of the specific template. A systematic study of different templates is an excellent direction for future work, and we will add this as a discussion point in our limitations section.

5. Q3 (Sufficiency of a Graph-only Pipeline):

We would like to clarify that a graph-only pipeline is not applicable for the tasks we address. Table question-answering requires understanding a natural language question. A pure graph-based model cannot process this textual query. The LLM is therefore an indispensable component for its language comprehension and reasoning abilities. Furthermore, we wish to clarify that the models mentioned by the reviewer, GCondNet and IGNNet, are designed for tabular prediction tasks (i.e., predicting a label for each row based on its features). This is a different task from Table QA, which involves generating a natural language answer based on a table and a question. Therefore, these models are not applicable baselines for our work.

6. Q4 (Hypergraph Losing Semantic Understanding):

This observation is correct and highlights the synergy of our approach. The hypergraph encoder is primarily designed to capture structural and relational properties. The serialized text, processed directly by the LLM, remains crucial for a deep, contextual understanding of the cell contents and the question. The two modalities are thus complementary.

We sincerely thank the reviewer for their valuable feedback and for acknowledging the importance of our work.

Comparison with Advanced Models:

We completely agree with the reviewer that evaluating on the latest models is crucial in this fast-paced field. To thoroughly address this critical point, we allocated significant additional computational resources to conduct comprehensive new experiments, including on several models accessible only via paid APIs.

First, to test whether the problem we identified still exists in the latest models, we evaluated several state-of-the-art LLMs you suggested on our StructQA benchmark. The results are as follows:

These results are very telling. Even after more than a year of rapid progress, today's most powerful LLMs still struggle with table structure understanding, a task that is trivial for humans. While their performance on the original StructQA test set has improved (reaching ~60%), every single model shows a significant performance degradation after permutation. This confirms our central hypothesis: the serialization of tables into text sequences leads to a loss of critical structural information, a problem that persists even in the latest SOTA models. This finding significantly strengthens the motivation for our work.

Second, to demonstrate that our proposed method, TAMO, remains effective on newer models, we conducted further experiments. Our original choice of a 7B-scale model was to ensure a fair comparison with the established TableLlama baseline. In addition to the LLaMA 3.1 8B results already in our Appendix C.6, we have now evaluated TAMO on the recent Qwen2.5-7B-Instruct and Qwen3-8B models. The results are compelling:

As shown, TAMO provides a dramatic performance boost on these new models. For instance, on Qwen2.5-7B, TAMO improves the accuracy on StructQA from 45.53% (Prompt Tuning) to 72.27%, an absolute improvement of nearly 27 points. This clearly demonstrates that our method of treating tables as a distinct modality is a powerful and relevant solution for enhancing the structural reasoning capabilities of even the most recent LLMs.

We will prominently feature these new findings in the revised manuscript. Thank you again for pushing us to strengthen our work in this crucial dimension.

Thank you for your detailed response and additional experimental results, comparing with current models. The additional comparisons are only provided for StructQA, the dataset that you developed, but not for other public datasets included in your paper. Would you be able to provide a comparison for at least a couple of the public datasets, comparing your method at least to GPT4.1? Presenting an improvement over current models only for one dataset that you developed seems less significant than showing it for some existing datasets.

Thank you for this crucial follow-up; it's an excellent point that pushes us to clarify the significance of our work. We agree completely and have conducted the extensive benchmarking experiments you requested. We respectfully clarify that our original evaluation philosophy was not to claim our 7B model surpasses foundation models orders of magnitude larger, but rather to demonstrate two key points: (1) structural reasoning challenges persist even in advanced models, and (2) our architectural innovation enables much smaller models to become highly competitive.

The results validate our approach. TAMO significantly outperforms all advanced models on StructQA (+11.27) and WikiSQL (+14.69), achieves state-of-the-art on HiTab, and remains competitive on FetaQA. WikiTQ's general-knowledge focus favors larger models' parametric knowledge, but TAMO's strong performance across four diverse benchmarks demonstrates broad applicability.

We hope these comprehensive results address your concerns and demonstrate that our architectural innovation enables accessible models to achieve remarkable structural reasoning capabilities, highlighting the real-world value of our contribution. Thank you again.

We thank the reviewer for their positive feedback on our work's innovation and clear logic. We address the thoughtful questions below.

1. W1 (Information Redundancy of Using Both Serialized Text and Structure Embeddings):

This is an excellent question regarding our design philosophy. We retain the serialized text because it provides the fine-grained semantic content of the table, which LLMs are exceptionally skilled at processing. The hypergraph-based structure embedding, conversely, provides the global relational structure (e.g., row-column relationships, hierarchies) that is inherently lost during serialization.

The two modalities are complementary: the text provides the content ("what"), and the structure embedding provides the context and relations ("where"). As our case study in Figure 5 suggests, the structure token helps the LLM better focus its attention on the relevant parts of the serialized text to locate the correct answer. Therefore, this dual representation is not redundant but synergistic.

2. W2 (Comparison with Other Multimodal Solutions like DocLLM):

We thank the reviewer for raising this point. We wish to clarify a fundamental distinction in the problem setting. Models like DocLLM or vision-language models (e.g., Qwen-VL) are designed to understand tables presented as images within documents, focusing on layout and OCR challenges. Our work, TAMO, addresses a different and complementary problem: reasoning over pre-structured tabular data (e.g., from databases, CSV files). As stated in our Limitations (Section 5), TAMO operates on this structured data format. The modalities are different (image+text vs. graph_structure+text), and the core tasks are distinct. Therefore, a direct comparison would not be appropriate.

We sincerely thank the reviewer for their valuable time and constructive feedback. We appreciate that the reviewer found our paper well-written and easy to follow. We address the concerns below.

1. W4 / Q1 (Clarification of "Specialist SOTA"):

This is an excellent point. The term is indeed vague. In our revision, we will update the caption of Table 2 to explicitly name the specific model and provide its citation for each dataset. The specific models are:

• For HiTab, the Specialist SOTA is TableLlama, which is based on Llama2-7B and underwent extensive supervised fine-tuning (SFT) on the HiTab training set along with other table QA datasets. The score reported in the original paper is 64.71, which we reproduced as 63.76.

• For WikiTQ and FetaQA, the SOTA is CABINET, which improves LLM performance by using an external scorer to assign higher weights to question-relevant table cells, thereby filtering out noisy ones.

• For WikiSQL, the SOTA is SeaD, which enhances text-to-SQL capabilities through a schema-aware denoising objective.

It is important to note that our purpose in including these specialist SOTAs is to highlight different methodological directions. Unlike our TAMO framework, which directly enhances the LLM's end-to-end understanding of the table as a modality, methods like CABINET and SeaD rely on external modules or task-specific pre-processing.

2. W5 / Q2 (Performance on WikiTQ vs. GPT-4):

We thank the reviewer for this sharp observation. We will add a discussion of this point to Section 3.3. Our analysis is twofold. First, the WikiTQ dataset is sourced from early Wikipedia web pages, and its questions are often of a general nature. This creates a potential for data contamination, where a model as extensively trained as GPT-4 may have already encountered these tables or facts during its pre-training, allowing it to answer some questions without fully relying on the provided table.

Second, as we initially noted, the performance gap is also attributable to the immense scale and powerful general reasoning capabilities of the proprietary GPT-4 model. However, our key finding remains valid: TAMO provides a substantial boost over its base model (Llama2-7B SFT: 45.81 vs. Text-only SFT: 31.22), proving the effectiveness of our approach. Furthermore, as shown in Appendix C.6 (Table 7), when TAMO is applied to a more capable base model like LLaMA 3.1 8B, its performance on WikiTQ improves significantly, demonstrating that TAMO's benefits scale with the underlying LLM's power.

3. On Comparisons with TAPAS/TAPEX (W3, Q3):

We agree that comparing with seminal structure-aware models like TAPAS and TAPEX is important. Our initial focus was on the modern decoder-only LLM-based paradigm, thus we prioritized comparisons with methods that also use serialized inputs (e.g., GPT-4, TableLlama). To further strengthen our evaluation, we have conducted additional experiments comparing TAMO with TAPAS and TAPEX on our StructQA benchmark. The results are as follows:

The results clearly show that TAMO dramatically outperforms these baselines on the core structural understanding tasks (StructQA and Permutation Accuracy). It is crucial to consider the architectural differences: TAPAS is a BERT-based encoder, and TAPEX is a BART-based encoder-decoder model. In contrast, our TAMO framework is specifically designed as a multimodal extension for decoder-only LLMs, which represent the current dominant architecture. While TAPAS and TAPEX show higher robustness, this is largely an artifact of their very low base accuracy; it is easier to remain consistent when a model is consistently incorrect. TAMO's performance demonstrates a far superior grasp of table structure. We will add a new table and this analysis to the appendix in our revised manuscript.

4. On Ablation Studies (W6, Q4):

We would like to respectfully clarify that our main experimental design in Table 2 constitutes a core ablation study. Across four different settings (Inference-only, Prompt Tuning, LoRA, SFT), we directly compare the performance of using text-only inputs versus our TAMO approach, which adds the table structure modality. The consistent and significant gains demonstrated across all settings (e.g., +21.27 absolute accuracy on StructQA for Prompt Tuning vs. TAMO) serve as the primary evidence for our method's contribution. We also provide more fine-grained analyses in Section 3.6 (encoder effectiveness) and Appendix C.3 (impact of token numbers).

5. On Novelty vs. HyTrel and Clarity (W2, Q5, Q6):

We appreciate the request for clarification. We respectfully point the reviewer to Appendix D.1, "Positioning of TAMO", where we provide a detailed discussion and a summary table (Table 8) to articulate the fundamental difference. In short, HyTrel is a modality encoder for learning table representations, whereas TAMO is a complete multimodal reasoning framework that integrates such an encoder into an LLM to solve joint text-table tasks (e.g., QA). This is a non-trivial conceptual leap, analogous to the distinction between ViT (an encoder) and LLaVA (a multimodal LLM). To ensure this crucial point is not missed, we will add a forward-reference to Appendix D.1 in our main methodology section. We will also revise Section 2.2 to further clarify the mathematical notations to improve readability.