Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models

We appreciate your thorough review and useful comments, particularly highlighting that our method is "novel and interesting application", and that our writing is "clear and easy to follow". Please find our responses below.

Weakness 1: More experiments are needed to demonstrate the method's effectiveness, potentially comparing it against closed-source models like GPT-4o with optimized prompting strategies.

Thank you for pointing this interesting question out. In the original evaluation of GPT-4o, we used the following prompt:

"<image>\nConvert this table to LaTeX. 
Please output only the LaTeX code between \\begin{tabular} and \\end{tabular}, without any additional text or explanations."

As suggested We then provided a more detailed prompt tailored to the table image-to-LaTeX task, explicitly indicating the potential complexity of the table structure and including an example LaTeX output. The specific prompt is as follows:

example_latex = r"""
Here is an example of a valid LaTeX table:
\begin{tabular}{|c|c|c|}\hline
Item & Quantity & Price \\
\hline
Apple & 2 & \$1.00 \\
Banana & 3 & \$0.75 \\
\hline
\end{tabular}
""".strip()
prompt_text = f"""{question}
Please extract only the LaTeX table between \\begin{{tabular}} and \\end{{tabular}} from the image.
The table may contain multi-row or multi-column cells, such as \\multirow or \\multicolumn.
Ensure the output has correct alignment and consistent row/column structure.
Do not include explanations or any extra text — only return valid LaTeX code for the table.
{example_latex}"""

However, this did not lead to improved performance. We believe this is because the task itself is clearly defined, and the initial prompt was already sufficient for GPT-4o to understand the objective.

Furthermore, we explored an iterative refinement strategy. Specifically, we fed the model the target table image, the initial LaTeX output, and the corresponding rendered image. The model was then asked to compare the rendered image with the target image—if no errors were detected, it would return the original LaTeX; otherwise, it would correct the LaTeX and return the updated version.

The specific prompt is as follows:

prompt_text = f"""The first image is the target table that should be recognized.
The second image shows the rendered LaTeX table from the previous prediction.

Here is the previous LaTeX prediction:
\\begin{{tabular}}{{...}}
{prediction_latex.split("\\begin{tabular}")[-1].split("\\end{tabular}")[0].strip()}
\\end{{tabular}}

Please compare the rendered prediction with the target image.
If the LaTeX code is already accurate, return it unchanged.
Otherwise, correct the LaTeX code.

Please output only the LaTeX code between \\begin{{tabular}} and \\end{{tabular}}, without any additional text or explanations."""

Table R1 presents the performance of different prompting strategies. The iterative refinement approach yields a marginal improvement in CW-SSIM and achieves a perfect Compile Ratio of 1.0. However, slight declines are observed in TEDS and TEDS-Structure metrics. We attribute this to the model focusing on visual similarity during the correction step, while lacking the ability to explicitly compare LaTeX structural syntax, which may have affected textual structure alignment.

TableR1: Ablation experiments on the effectiveness of different prompt for GPT4o.

Weakness 2: Are tables with annotations (e.g. bold, red mark, color highlight) considered in the dataset and evaluation? How do they perform with the proposed method?

To further validate the effectiveness of our method as suggested by the reviewer, we additionally evaluated it on two publicly available table datasets: FinTabNet [1], which comprises real-world, complex scientific and financial tables with detailed structural annotations, and SynthTabNet [2], which includes Chinese financial announcement tables. A human evaluation was conducted to assess the visual similarity between the LaTeX-rendered images generated by our method and those produced by existing state-of-the-art MLLMs, using the original table images as ground truth.

We randomly sampled 100 images from the validation set. For each sample, the ground truth table image is displayed alongside the model's predicted table image, with their positions randomly shuffled and labels hidden. Multiple human assessors independently voted on which image was more visually similar to the target, with the final outcome determined by majority vote. As shown in Tables R1 and R2, our method consistently demonstrates competitive or superior performance across both datasets.

Table R1：Results of human evaluation in FinTabNet[1].

Table R2：Results of human evaluation in SynthTabNet[2].

Reference

[1]Zheng, Xinyi et al. “Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020): 697-706.

[2]Hou, Qiyu et al. “Synthesizing Realistic Data for Table Recognition.” IEEE International Conference on Document Analysis and Recognition (2024).

Question 1: For Tables 1-3, how is GPT-4o evaluated? Could a refined prompting strategy, such as specifying table complexities (e.g., multirow, multicolumn), providing examples, or using iterative correction, yield better LaTeX generation results?

See above, in "Weakness 1".

Question 2: Are tables with annotations (e.g. bold, red mark, color highlight) considered in the dataset and evaluation? How do they perform with the proposed method?

See above, in "Weakness 2".

Dear Reviewer, thank you for your response, your continued engagement, and your commitment to improving our paper! Please find our responses below.

Weakness 1: The main concern is the limited technical novelty, as the proposed LLM fine-tuning and RL solution is standard, with the only notable contribution being the reward design.

Thank you for acknowledging the practical value of our task and the effectiveness of our method. However, we respectfully disagree with the assessment that our work lacks technical contribution. Notably, all three other reviewers explicitly recognized the novelty and technical strength of our approach:

• Reviewer A7cA stated: “The proposed approach is novel.”

• Reviewer 6PYg commented: “The idea of combining visual and structural reward is interesting.”

• Reviewer e8fZ noted: “Novel and interesting application to leverage MLLM’s ability to address complex table LaTeX generation.”

These consistent positive remarks affirm our view that the paper presents meaningful and original contributions.

1. Revealing MLLM Capability and Proposing an Effective Solution

The task is underexplored and one of our contributions lies in demonstrating for that MLLMs can effectively handle table image to LaTeX generation—a task that is highly structured, syntactically complex, and not traditionally studied in the vision-language domain. We establish a complete and scalable MLLM-based training pipeline, incorporating large-scale supervised fine-tuning (SFT), and reinforcement fine-tuning (RFT). This pipeline shows that MLLMs, when properly guided, can directly generate high-quality LaTeX code, revealing their potential in structured code generation tasks previously considered beyond their typical scope.

2. Technically Novel Reward Design for RL Fine-tuning

From a technical perspective, our proposed Visual-Structural Guided RL (VSGRPO) framework offers a novel and practical advancement. Unlike existing RL strategies that rely on rule-based textual matching, which are often brittle or ineffective for long, complex outputs like LaTeX, we propose a dual-reward system:

• A visual reward, which evaluates the fidelity of the generated LaTeX by rendering and comparing it to the ground-truth image—providing a more direct and perceptually aligned quality measure.
• A structural reward, which ensures syntactic and layout consistency in the generated code. This dual-reward mechanism not only improves performance on our target task but also has the potential to generalize to other vision-to-code generation problems, such as web UI generation or document formatting.

Weakness 2: There are very few qualitative figures in this paper.

Thank you for pointing this out. We agree that qualitative results are important for providing intuitive insights into model behavior.

In our paper, we divided the evaluation into simple, medium, and complex tables, and provided four representative sets of qualitative visualizations in the Appendix, covering all three levels of table complexity. These examples were selected to illustrate typical error patterns among the compared methods, helping readers better understand the nature of improvements introduced by our approach.

We acknowledge that four examples are not sufficient to comprehensively showcase all aspects of model performance. However, due to strict page limitations, including a larger number of qualitative examples in the main paper was not feasible.

That said, we would like to emphasize that our evaluation is not solely reliant on qualitative examples. We have also included:

• Extensive quantitative comparisons across multiple benchmarks.

• A scientifically designed and rigorously conducted human evaluation.

• Together, these three forms of evaluation—quantitative, qualitative, and human—consistently demonstrate the advantages of our proposed method over existing baselines.

Weakness 3: The effectiveness of the introduced two rewards is not significant as shown in Tab. 6.

Thank you for your feedback. We emphasize that the visual and structural rewards are crucial for improving model performance, both quantitatively and perceptually.

First, as shown in Table 6, our dual-reward mechanism significantly improves CW-SSIM during reinforcement learning, boosting scores by 3.5 and 2 points over no RL and text-only rewards respectively. It is important to note that even a 2-point increase in CW-SSIM is non-trivial and translates into visibly better table rendering quality, as confirmed by human visual inspection.

Second, to validate these gains from a human perspective, we conducted a blind human evaluation on 100 randomly selected complex table samples. Annotators compared the generated images to ground-truth images in randomized order, without knowing the source model. As summarized in Table R1, the dual-reward VSGRPO model was preferred 83 times, significantly outperforming the single-reward variants (Visual-only: 59, Structure-only: 44).

These results confirm that combining visual and structural rewards leads to more faithful and visually accurate LaTeX table generation.

Table R1: Results of human evaluation in the two rewards.

Question 1: Directly using the continuous similarity value as a reward would be more effective.

Thank you for this insightful comment. We indeed explored this direction by conducting an ablation study comparing continuous rewards (based on CW-SSIM values) with binary rewards (based on thresholding) using the Qwen2.5-VL-3B model. The results are summarized in TableR2:

TableR2: Ablation experiments on the effectiveness of continuous reward value.

As shown, the continuous reward leads to slightly inferior performance. One possible explanation, supported by [1], is that higher reward variance can lead to stronger learning signals in RL. Binary rewards, especially when carefully thresholded, may create sharper gradients and more decisive feedback, which can help the model escape suboptimal regions in the reward landscape. In contrast, continuous rewards, being bounded and smoother, might lack this reinforcement strength. This remains an open research question, and we believe further investigation into the design of reward functions could be a promising direction for future work.

Reference

[1]Razin, Noam et al. “What Makes a Reward Model a Good Teacher? An Optimization Perspective.” arXiv abs/2503.15477 (2025): n. page.

Question 2: The "w/o SFT" model's worse performance in Tab. 7 might be due to a smaller training sample size (5,936 vs. full version), rather than solely indicating SFT's necessity.

We would like to clarify that the same number of training samples was used in both the "with SFT" and "without SFT" settings. As stated in Line 169, “we select 5,936 complex tables from the training dataset as the training set for VSGRPO.” This exact subset was used consistently across both settings during the reinforcement learning (RL) stage.

Therefore, the difference between the two methods in Table 7 reflects only the presence or absence of the SFT stage. The performance gap thus directly highlights the importance of SFT in preparing the model for effective RL fine-tuning.

We also note that fewer samples are used in RL than in SFT due to the higher computational complexity of the RL process.

Question 3: The performance gains from VSGRPO in Tab. 6 appear marginal; a discussion from the authors on this would be beneficial.

See above, in "Weakness 3".

Question 4: The Abstract and Introduction emphasize Table2LaTeX's challenges—large, nested, and semantically rich tables—yet the Method section lacks specific techniques to address them, creating a disconnect for the reader.

Thank you for pointing out this concern. We understand the reviewer’s impression and would like to clarify that the methodological components in our work are in fact carefully designed to address the core challenges highlighted in the Abstract and Introduction—namely, large table sizes, deeply nested structures, and semantically rich content.

As the reviewer noted, this is a practically valuable but underexplored task, and part of our contribution lies in articulating its unique difficulties. To effectively tackle these, we built a training pipeline that combines the strengths of Multimodal Large Language Models (MLLMs) with domain-specific enhancements:

• Large-scale supervised fine-tuning (SFT) on diverse and richly annotated table–LaTeX pairs, enabling the model to generalize across various table structures and adapt to the specifics of the Table2LaTeX task. This step is critical for equipping the model with robustness in handling a wide range of real-world table layouts.

• Reinforcement Learning with Complex Examples: During RL, we intentionally select complex tables—those with large size, deep nesting, and rich semantics—as training samples. This design ensures the model is directly optimized for the most challenging aspects of the task. This design achieves superior performance than using.

We thank the reviewer for their comments, especially for highlighting our paper could have "important application" and that our idea is "interesting ". Below, we address the issues raised in the review:

Weakness 1: The paper cites LATTE[1] and reports its results, but lacks analysis comparing dual-reward RL to LATTE’s approach, limiting insight into VSGRPO’s motivation and novelty.

Thank you for raising this important point.

First, we would like to clarify that LATTE has not released its full implementation, particularly the code for the iterative refinement module. This component depends on a separately curated dataset, which is also not publicly available. As a result, we are unable to precisely reimplement LATTE or conduct a direct comparison beyond referencing the limited results provided in their paper.

Second, from a methodological perspective, our approach and LATTE differ fundamentally in design philosophy and execution. Our method adopts the LLM/MLLM paradigm by establishing an end-to-end pipeline encompassing large-scale supervised fine-tuning (SFT), and reinforcement fine-tuning (RFT) for the table-to-LaTeX task. In contrast, LATTE centers around an iterative post-refinement strategy, which is applied after initial LaTeX generation.

In other words, our goal is to embed high-quality LaTeX generation capabilities directly into the model, enabling accurate outputs at inference time without needing further correction. LATTE, on the other hand, relies on a separate, explicitly designed refinement stage to improve the quality of the initial outputs, which introduces an additional computational overhead and complexity during deployment.

These distinctions reflect different design goals: our method aims for integrated accuracy and efficiency, while LATTE emphasizes post-hoc quality improvement through refinement.

Weakness 2: Though cited, vision-only models like [2] aren’t evaluated, leaving unclear how the proposed method compares to established CV approaches.

Thank you for bringing this up. Our method differs fundamentally from traditional computer vision (CV) approaches by leveraging the text generation capabilities of Large Language Models (LLMs) rather than relying on rule-based decoding pipelines.

Traditional CV solutions like TableFormer primarily target HTML table reconstruction, which benefits from a relatively rigid and sequential syntax. These methods typically decompose the problem into two steps: first detecting the table layout, then predicting the cell content, following the HTML structure where each cell is enclosed in <td> </td> tags. The deterministic nature of HTML allows such systems to adopt rule-based strategies for both structure and content prediction.

In contrast, our task focuses on LaTeX table generation, which presents significantly greater complexity. LaTeX syntax is more flexible and less strictly structured than HTML, making it challenging to apply rule-based layout parsing or sequential tagging approaches. As such, we propose to treat table-to-LaTeX as a generation task, enabling LLMs to directly produce high-fidelity LaTeX code in a more holistic and end-to-end fashion. This not only aligns better with the open-ended nature of LaTeX syntax but also takes advantage of the strong reasoning and generation capabilities of modern MLLMs.

Weakness 3: Training on mostly simple arXiv tables (94%) limits insight into the model’s robustness to scanned or colored documents.

As rightfully requested by the reviewer, we have used two additional publicly available table datasets to further validate the effectiveness of our method: FinTabNet [3], which consists of real-world and complex scientific and financial tables with detailed structure annotations, and SynthTabNet [4], a dataset of Chinese financial announcements. We conducted a human evaluation to compare the visual similarity between the LaTeX-rendered images generated by our method and those from existing state-of-the-art MLLMs, against the ground truth table images.

We randomly selected 100 samples from the validation set. For each sample, the ground truth table image and the model's predicted table image were displayed side-by-side. The display order was randomly shuffled and labels were hidden to ensure an unbiased assessment. Multiple human assessors independently cast their votes for the image most visually similar to the target, with the majority vote determining the final result. As shown in Tables R1 and R2, our method also demonstrates consistently strong competitiveness in these baselines.

Table R1：Results of human evaluation in FinTabNet[3].

Table R2：Results of human evaluation in SynthTabNet[4].

Reference

[1] Jiang, Nan, et al. "LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.

[2] Nassar, Ahmed, et al. "Tableformer: Table structure understanding with transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3]Zheng, Xinyi et al. “Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020): 697-706.

[4]Hou, Qiyu et al. “Synthesizing Realistic Data for Table Recognition.” IEEE International Conference on Document Analysis and Recognition (2024).

Question 1: Can the authors provide analysis or ablation to show why dual-reward RL outperforms LATTE and clarify VSGRPO’s motivation and novelty?

See above, in "Weakness 1".

Question 2: Can the authors evaluate vision-only models or explain why they’re unsuitable to clarify how their method compares to mature CV pipelines?

See above, in "Weakness 2".

Question 3: Can the model trained on arXiv tables generalize to other domains like finance, e.g., via case studies on FinTabNet?

See above, in "Weakness 3".

Dear Reviewer, thank you for your thorough review and helpful suggestions! We also appreciate you mentioning that our paper has "important application value" and our approach is "novel". Please find our responses and changes below.

Weakness 1: The results are biased since both training and test data come from arXiv tables, making it unfair to fine-tune models like Intern2-VL-VSGRPO and compare them with baselines on in-domain tests.

We appreciate the reviewer’s thoughtful concern regarding the evaluation fairness and potential in-domain bias stemming from training and testing on LaTeX tables sourced from arXiv. In response, we have extended our evaluation to include two additional publicly available out-of-domain table datasets: These two datasets, primarily derived from financial documents and reports, are markedly out-of-domain relative to arXiv LaTeX tables. They differ in content domain, table styles, layout complexity, visual noise and even the presence of colored cells, providing a challenging and comprehensive testbed for evaluating model generalization. FinTabNet [1], which consists of real-world and complex scientific and financial tables with detailed structure annotations, and SynthTabNet [2], a dataset of Chinese financial announcements. Unlike the arXiv-based LaTeX tables, these datasets are typically provided in HTML or image format without LaTeX source code. Therefore, CW-SSIM scores cannot be computed, as they require image rendering from ground-truth LaTeX for comparison. To address this, we conducted a human evaluation study, comparing the visual fidelity of LaTeX-rendered tables generated by our method against state-of-the-art MLLMs, using the original table images as ground truth references.

We randomly selected 100 samples from the validation set and display the ground truth table image at the top and the model’s predicted table image below it, side by side, with the order randomly shuffled and labels hidden. Multiple human assessors independently vote on which image appears most visually similar to the target. The final outcome is determined by majority vote. As shown in Tables R1 and R2, the results of this human study demonstrate that our method consistently achieves superior visual quality across both datasets, indicating its strong generalization capability to tables from different domains and formats. These findings provide further evidence of the robustness and practical applicability of our approach beyond the arXiv domain.

Table R1：Results of human evaluation in FinTabNet.

Table R2：Results of human evaluation in SynthTabNet.

Reference

[1] Zheng, Xinyi et al. “Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020): 697-706.

[2]Hou, Qiyu et al. “Synthesizing Realistic Data for Table Recognition.” IEEE International Conference on Document Analysis and Recognition (2024).

Weakness 2: VSGRPO vs. w/o using VSGRPO.

Thank you for this insightful suggestion. We agree that a direct comparison between our method with and without VSGRPO provides a clearer understanding of its contribution. To this end, we conducted a focused human evaluation on 100 randomly selected samples, comparing the full model (with VSGRPO) against its ablated variant (without VSGRPO). The results are as follows:

• VSGRPO was preferred in 52 cases.

• The variant without VSGRPO was preferred in 18 cases.

• 30 cases were judged as indistinguishable in quality.

These results demonstrate a clear preference for the VSGRPO-enhanced version. In addition, as part of our out-of-domain evaluation (detailed in response to the previous comment), we also performed comparisons between the two variants on external datasets (FinTabNet and SynthTabNet). The outcomes consistently showed that the VSGRPO-enhanced model produces LaTeX-rendered table images with higher visual similarity to the ground truth, further validating its effectiveness.

Weakness 3: VSGRPO isn’t compared with existing RL methods like GRPO, making it hard to show its advantage.

We appreciate the reviewer’s suggestion and would like to clarify that we have included a direct comparison with GRPO as a baseline in our paper. Specifically, the original GRPO framework utilizes a rule-based verifier to assess the quality of generated text against the ground truth. In the context of table generation, this would involve comparing the generated LaTeX code with the ground-truth LaTeX using a manually designed verifier. However, unlike typical question-answering tasks where the ground truth is relatively short and correctness is easier to verify, LaTeX code is often long, structurally complex, and challenging to evaluate using exact matching or heuristic rules. To adapt GRPO to our task fairly and meaningfully, we replaced the original rule-based verifier with a TEDS-Structure-based verifier, which is better suited to assessing structural and visual similarity between table representations. The performance of this adapted GRPO variant is reported in Table 6 as Qwen2.5-VL-3B-GRPO-TEDS-Structure. As shown in the results, our proposed VSGRPO significantly outperforms this GRPO baseline, highlighting the effectiveness of our framework in handling the structured nature and generation complexity of LaTeX tables.

Question 1: Does using only the outcome reward yield similar gains, e.g., Intern2-VL-1B-GRPO vs. VSGRPO?

See above, in"Weakness 3".

Dear Reviewer,

As the discussion period is coming to a close, we wanted to kindly follow up. We have provided a comprehensive response to your comments and hope that we have adequately addressed your concerns. If you have any further questions or would like to discuss any aspect in more detail, we would be very happy to continue the discussion.

Thank you again for your valuable feedback.

We appreciate the opportunity for this discussion. As the reviewer-author discussion comes to a close, we note that the reviewer has not yet responded to our original rebuttal. We believe we have carefully addressed the concerns raised and have provided clarifications and additional evidence where needed. We respectfully request that the Area Chair carefully review our responses and supporting materials when making the final decision.