Compositional Condition Question Answering in Tabular Understanding

We appreciate your recognition of our setting and method and your valuable comments. We will answer the questions below, and we hope this clears up your concerns.

Q1: additional computational overhead

A1: We appreciate the need to rigorously evaluate computational efficiency. While our paper focused on accuracy and structural alignment improvements, we conducted additional experiments to quantify the computational overhead of CoCoTab. Here are the key results and analyses:

We have measured the average inference time and memory consumption per sample. As shown in the results, our method does not introduce significant overhead. On the contrary, compared to LLaVA, our approach provides more concise and precise answers, leading to improved inference speed. As noted in response A2 to Reviewer eFpb, COCOTAB adds only 7% in token count but outperforms in all baselines.

Q2: More baselines and results.

A2: Thank you for your critical feedback. We appreciate the opportunity to clarify and strengthen our experimental comparisons. Below, we address each concern systematically:

Our work focuses on document and tabular understanding, a subfield of MLLM that demands specialized structural alignment capabilities. To ensure fair and state-of-the-art comparisons, we selected the strongest open-source baselines published in 2024, including LLava1.6(baseline), TextMonkey, DocOwl2.These models represent the most advanced techniques in document/table understanding as of mid-2024.

For completeness, we also compared it with InternVL2.5-8B and TabPedia (NeurIPS 2024).

Although our method may not outperform InternVL and TabPedia in certain aspects, it significantly surpasses both in CC performance. Moreover, our training data (1.8M) is considerably smaller than that of TabPedia (~4M) and certainly much less than InternVL. Despite this, our approach still achieves competitive performance. We believe these results conclusively demonstrate CoCoTab's advancements in tabular understanding and we promise we will add these comparisons in our final version.

Q3: The improvement on weak baseline LLaVA.

A3: Thank you for raising this critical point. We appreciate the need to validate our method’s contributions against stronger baselines and would like to clarify our experimental design:

Our ablation studies (Section 5.2) intentionally used the original LLaVA architecture (Vision encoder + MLP + LLM) to isolate the impact of our proposed components (row/column patches and conditional tokens) under strictly controlled conditions. By keeping the LLM, training data, and hyperparameters identical, we ensure that performance differences directly reflect the effectiveness of our structural enhancements. This approach is standard in ablation studies to disentangle contributions from architectural changes versus other factors. Our ablation study shows that the performance gain validates the importance of our proposed components.

From another perspective, our ablation baseline (LLaVA + our training data) already achieves higher accuracy than vanilla LLaVA-1.6, demonstrating that our curated training data itself improves performance. This validates the quality of our data pipeline even before adding structural components.

Q4: How the image patches with the size of HP and PW are input into the image encoder.

A4: We appreciate the opportunity to clarify our methodology for handling multi-scale patches.

In our CoCoTab, we generate row-wise (HP) and column-wise (PW) patch embeddings in parallel with the standard PP patches using lightweight convolutional operations. Here are the key steps:

The input image is resized to 384×384 (as in SigLIP) and split into 27×27 patches of size 14×14 (P=14) with a 2D convolutions with kernal size PP, following standard ViT processing. Similarly, we apply 2D convolutions with kernel sizes H×P (for rows) and P×W (for columns) to the resized image. These operations are computationally lightweight and performed before the ViT’s patch embedding layer. This design allows the ViT to implicitly reason about row/column structures without violating its architectural constraints. We will revise Section 4.1 to detail these steps.

Thank you again for highlighting this ambiguity. We hope this clarification helps strengthen the details of our proposed CoCoTab.

We appreciate your recognition of our setting and method and your valuable comments. We will answer the questions below, and we hope this clears up your concerns.

Q1: A well-structured table might not further transfer to a more generic type of vision input.

A1: Thank you for this insightful observation. We fully agree that extending tabular understanding to unstructured or free-form visual inputs (e.g., handwritten blackboard tables) is an important and challenging direction. Our work focuses on structured tables for two key reasons:

Foundation for Real-World Applications: Existing benchmarks (e.g., WTQ, TabFact) and practical use cases predominantly involve structured tables. Even in these "well-structured" settings, current MLLMs struggle with basic compositional reasoning. Addressing these fundamental limitations is a necessary step before extending to more complex, unstructured scenarios.

Generalization via Training Diversity: While our primary evaluation focuses on structured tables, CoCoTab’s training incorporates diverse visual QA datasets (e.g., OCR-VQA, PlotQA) that include semi-structured inputs such as book covers, posters, and diagrams. These datasets help the model develop robustness to irregular layouts, partial occlusions, and noisy text—capabilities that can facilitate generalization to blackboard-style tables.

We acknowledge that fully adapting to free-form tables requires additional innovations, such as dynamic patch resizing to accommodate arbitrary layouts. We will incorporate a discussion of these limitations and potential future directions in the final version.

Q2: More patches and the cost.

A2: Thank you for raising this practical concern. We acknowledge that introducing additional patches incurs computational overhead. However, our design carefully balances cost and performance:

The original method divides the table image into 27×27 patches (729 total). COCOTAB adds 27 row patches (one per row, covering full width) and 27 column patches (one per column, covering full height), resulting in only 54 additional patches (7% increase over the original 729). This modest increase in token count (from 729 to 783) incurs minimal computational overhead during training and inference.

As shown in our ablation study, adding row/column patches alone improves accuracy on CC tasks over the baseline. This demonstrates that the cost increase is highly justified by the performance gains, particularly for critical tasks like compositional reasoning.

In summary, COCOTAB achieves performance gain with only a marginal cost increase, making it a practical and efficient solution. Please refer to A1 to Reviewer qd5K for more details.

Q3: The baselines are slim.

A3: Thank you for your feedback. Our work focuses on improving tabular understanding in MLLMs, especially for compositional QA. To ensure fair comparisons, we chose state-of-the-art document-understanding MLLMs that integrate advanced techniques like data augmentation and positional encoding. For example, Donut: Uses synthetic data for pertaining. DocOwl: Leverages high-resolution document images for positional awareness. TextMonkey: Utilizes shifted window attention for spatial understanding.

These baselines represent the strongest open-source MLLMs in this area and inherently employ the techniques you mentioned. Yet, as shown in Table 2, CoCoTab outperforms them on all tasks, highlighting that existing approaches still struggle with tabular structure alignment and conditional reasoning. More experiments are in A2 to Reviewer qd5K.

Q4: The evaluation and comparison don't seem fair.

A4: We appreciate the importance of fair comparisons and would like to clarify our experimental design:

Strict Data Separation: While WTQ was included in our training data for Stage 1, we rigorously ensured that the test splits of WTQ, TabFact, and other benchmarks were never exposed during training. Many compared models are pre-trained or fine-tuned on the same data (e.g., DocOwl uses DocVQA, TabFact and WTQ). Our work follows a similar protocol but focuses on structural alignment.

MMTU: The proposed MMTU was designed to evaluate TQA from different aspects. It includes tables from WTQ but reformulates questions and introduces new compositional conditions, ensuring no overlap with training data. Previous work [1] shows that existing benchmarks lack variation in their questions, resulting in a bad performance with perturbations. CoCoTab's superiority on MMTU and ablation study further validates its generalizability.

In summary, our experiments adhere to fair evaluation standards, and the improvements are attributable to our method’s architectural innovations rather than data bias. We appreciate your attention and we will make it clear in the final version.

[1] CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs. NeurIPS, 2024.

We appreciate your recognition of our setting and method and your valuable comments. We will answer the questions below, and we hope this clears up your concerns.

Q1: While the paper emphasizes the vision encoder’s patch-based approach as a major contributor to misalignment, have you conducted experiments to isolate the impact of the LLM’s reasoning and alignment capabilities separately?

A1: Thank you for raising this critical question. We fully acknowledge the challenge of disentangling the LLM’s reasoning capabilities from alignment issues in multimodal systems. To address this, we designed ablation studies (Section 5.2 and Figure 5) to isolate the impact of our proposed structural enhancements (row/column patches and conditional tokens). These performance gains with our architecture were achieved without modifying the LLM, demonstrating that our structural enhancements directly address vision-text alignment limitations, independent of the LLM’s reasoning capabilities.

We agree that stronger LLMs can further improve performance. However, our method focuses on optimizing the visual encoder and cross-modal interaction, which are orthogonal to the choice of LLM. This ensures compatibility with any downstream LLM.

In summary, our ablation studies confirm that the performance improvements stem from better structural alignment in the visual encoder and cross-modal attention, rather than LLM-specific reasoning enhancements. Thank you for your valuable question. We promise that we will make this clear in the final version.

Q2: Did you have some qualitative examples showing the failure cases of COCOTAB?

A2: Thank you for this question. While COCOTAB improves performance on CC tasks, we acknowledge that failure cases persist, particularly in scenarios involving OCR errors or complex reasoning. Here are two representative examples from our analysis:

In our response in CC task, the model correctly identified relevant rows but erred in misrecognized scores (e.g., “93” misread as “98” in one cell). As shown in Appendix E, the OCR ability of our model may be strengthened through other strategies, such as more datasets and more vision encoders or tokens.

From another perspective, when it comes to CR tasks, the model erred in computing the difference between 63.49 - 38.77 (= 24.72) as 37.22. In this paper, we concentrate on CC tasks. For such calculation and reasoning tasks, other methods such as Group Relative Policy Optimization (GRPO) in DeepSeek will help improve performance.

Thanks for your valuable suggestion. We will discuss these failure cases in Appendix and plan to address them via OCR error correction and enhance the reasoning ability in future work.