General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model

Thank you very much for your review. We will do our best to address your concerns:

• After a year of rapid development, there is already a consensus that LVLMs have higher ceilings and potential for OCR compared to traditional OCR models (including traditional SOTA models). Therefore, in the limited paper space available, our experiments mainly focus on comparing the most advanced LVLMs. After the paper is made public, we are happy to conduct more comparisons and strive to promote open-source efforts to allow more people to evaluate the model’s performance across various subtasks.

  Additionally, it is important to note that this is a research paper, and we believe comparing commercial tools is inappropriate. Second, the model algorithm proposed in this paper should serve as a baseline for OCR-2.0 technology, and we hope this paper brings new research perspectives to the OCR field. GOT is not a tool, nor is it called “OCR anything,” so we cannot cover massive domain-specific tests, and excessive demands would be unreasonable. For the related work section, we will include more traditional specialized-task OCR models in the next version of the manuscript. Thank you.

• Thanks for your comments. The reason we did not test on the common standard datasets (natural images) is that our primary comparison targets are LVLMs, and we are unsure whether they have used the relevant data for training. Our natural image test data is sourced from the same origins as most LVLMs (such as Laion, Wukong, etc.), and this choice was made to ensure a fair comparison.

  Besides, we acknowledge the efforts and importance of previous OCR-related works and benchmarks, but we also believe that benchmarks will continue to change with the development of OCR technology. One thing that also must be acknowledged is that the classic benchmarks from the past are no longer suitable for evaluating the OCR capabilities of LVLMs, because previous benchmarks were designed for those OCR models with detection-cropping processes, and many images were in small cropped format. When we have OCR-2.0 models that can perceive text in entire images, we need to consider whether we should rigidly continue using previous benchmarks. We hope that the proposed new benchmark will also be one of the contributions of our paper.

• Our training is designed to save resources under large models and long token lengths, rather than to enhance performance. For example, in Stage 1, we use a smaller language model as the decoder to quickly warm up the encoder, which allows us to save about half of the resources. For excessively long texts, such as multi-page content or scenarios involving extremely long image crops, we place them in Stage 3, where we freeze the encoder to reduce the additional GPU memory costs associated with encoder activations.

  In summary, the value of GOT's training paradigm cannot be judged by previous approaches because earlier models did not handle extremely long tokens and did not have large parameter counts. The resource bottleneck problems they encountered were much smaller in scale.

[Question] Our evaluation metrics are common standards, and due to space limitations, we did not include them in the main text. The calculation methods for the chart metrics are referenced in our paper, such as OneChart (Chen et al.).

Sincerely hope that my response has resolved your concerns.

Thank you very much for your professional suggestions. We will do our best to address your concerns:

[Weakness]

• Thank you for the good suggestion. Changing “theory” to “technology” is a good idea. It will not alter the model’s abbreviation, GOT. We will consider your suggestion in the next manuscript. Thank you once again!
• After discussion, we have reached a consensus that after GOT is successfully published, we will open-source most of the training data to further promote community development. We maintain a very open attitude towards open-source and will continue to optimize this GOT open-source-plan in the future, enabling everyone to better understand and reproduce the model.
• The reason we did not test on the dataset you mentioned is that our primary comparison targets are LVLMs, and we are unsure whether they have used the relevant data for training. Our natural image test data is sourced from the same origins as most LVLMs's OCR data (such as Laion, Wukong, etc.), and this choice was made to ensure a fair comparison.

Regarding the dataset you mentioned, most of them were not used during our training. One reason is the relatively small data volume, and another is that the annotation formats of each dataset differ. We were concerned that training them might overfit specific, non-generalizable annotations, which could affect the model’s data scaling-up ability. Fortunately, we can perform separate SFT for each dataset. We selected CTW1500 and ReCTS, and fine-tuned for just one epoch. Because GOT did not have detection capabilities, we split the output results by spaces and used the ‘include’ method to determine accuracy by checking if the output is in the ground truth. GOT achieved 83.9% and 84.7% accuracy on both CTW1500 and ReCTS, which I provide as a reference for you.

[Question]

• As mentioned above, the reason we did not evaluate certain public datasets is that we were uncertain whether other LVLMs had used them, and since each dataset has specific annotation formats, zero-shot evaluation was not advisable. To ensure a fair comparison, we also refrained from including most of these datasets in our training. A feasible approach is that we will include a comparative analysis of the supervised fine-tuning (SFT) performance of various LVLMs across the mentioned datasets in our future manuscript.
• Yes, all test datasets will be open-sourced. If the paper goes smoothly, we will also make our training data publicly available in the future.
• Thank you for your suggestions. We will correct some typos in our future manuscript.
• I strongly agree with the latter point of your suggestion about OCR-2.0 - high extensibility. Currently, we have successfully fine-tuned GOT across various languages, such as Arabic, Haitian, and Indian languages, which demonstrate extremely strong extensibility. Thus, we believe GOT will be an excellent baseline.

Sincerely hope that the response has resolved your concerns.

Thank you very much for your comments. We will do our best to address your concerns:

• For different tasks, routing through different prompts is necessary, which can be found in the paper and illustrated in the supplementary materials. More details can be found in the submitted codes.  In the future,  we will actively promote open-source to make the usage of GOT clearer.

• We would like to politely point out that the tasks you mentioned are not OCR tasks; they fall under VQA (Visual Question Answering) tasks. Even if they involve texts, text-driven VQA should not be regarded as OCR, i.e., high scores in VQA do not directly reflect a model’s OCR capabilities. These tasks are merely downstream applications of OCR, and OCR-2.0 is not designed for downstream tasks.

  For example, in DocVQA samples, OCR capabilities would involve transcribing the entire document in its original format, whereas QA does not necessarily require a strong perception of the entire text. In QA, learning certain answering patterns may allow the model to find correct answers by searching through parts of the text. Therefore, even text-dependent QA and OCR are not equivalent. We should not include these QA metrics. Thanks.

• The “R” in OCR stands for Recognition and should not be tied to detection. Previous methods required detection because they lacked strong perception and full-text recognition capabilities, and thus relied on detection boxes to crop specific areas for recognition. A key point of OCR 2.0 is the ability to perceive and recognize dense, full-page images without relying on detection. Regarding your concerns about the detection capabilities of auto-regression models, although it is unrelated to our model, we are willing to discuss this with you:

  We believe that there is an essential difference between the current LVLM paradigm and the original detection models for detection tasks. The LVLM’s approach of using next-token predictions to output text (string) for predicting numerical coordinates (float) is clearly at a disadvantage under the current training and testing paradigms. However, this does not mean that this architecture cannot perform well on this task. Referring to the process of human-annotated detection data, the current LVLM architecture should perform multiple rounds of visual inspection and box refinement for a target to achieve good results, which implies multiple times perceptions. This point is also applicable to the aforementioned VQA tasks. The current LVLM architecture is best suited for quick recognition, which is also the original intention behind promoting OCR 2.0.

Sincerely hope that my response has resolved your concerns.

Thank you for your review.

Based on your comments, we politely speculate that your background might be in the traditional Computer Vision domain of AI-1.0, and you may not be a frontline researcher in the LVLM-related field. Therefore, there might be some bias in your assessment of GOT’s contributions and innovations for LVLM-OCR field. We hope that we can address your concerns to the best of our ability.

[Highlight] First, we emphasize that the design of the GOT model is unique and innovative in both the fields of OCR and LVLMs. OCR performance is one of the most important capabilities in current LVLMs, and strong text perception serves as the foundational design principle for an LVLM. The introduction of GOT provides a new methodology for designing LVLMs, which is reflected in:

• A key highlight of the paper is that for LVLMs, performing dense perception tasks (such as document-level OCR) does not require as many image tokens (which is currently a bottleneck in LVLM design). GOT can decode images containing over 4000 words using only 256 image tokens, whereas other LVLMs waste thousands of tokens. In other words, for LVLMs, token density (= number of encoded pixels / visual tokens, proposed in MiniCPM) still has significant room for improvement:（GOT: 4096; MiniCPM2.6: 2822; GPT4o:1088; Qwen-VL-max:784; InternVL2: 706; LLaVA-Next:157)

• There may still be room to reduce the size of the decoder. We achieved a better general OCR performance than a model larger than 72B （Qwen-vl-max）using only 0.5B, providing a reference for designing small-scale LVLMs for edge devices.

[Training] With regard to model training, our training architecture also has valuable and guiding significance in the LVLM field. GOT’s training primarily focuses on how to save training resources, especially when training the encoder and expanding the max token length. Our aim is completely different from decoupled pre-training approaches (such as UniT, BLIP), because their parameters and max token length may not necessarily encounter the resource bottlenecks faced during LVLM training. The value of our training paradigm lies in:

• To achieve a model with high token density, there should not be a freeze LLM phase in the entire LVLM training process. This is the approach taken by many existing LVLM models, such as BLIP-2, LLaVA, Qwen-VL, and mPlug-Owl in their stage-1. Instead of directly aligning image tokens to the LLM, the tokens must be aligned with each other to obtain high compression decoding ability for text.
• The above settings will inevitably significantly increase the required resources. GOT’s stage-1 aims to use a small language model to train an encoder suitable for LLM under lower resource conditions, with the key point absolutely not being decoupling pretraining (you said UniT, BLIP). GOT’s stage-2 is designed to train data with diversity and shorter tokens, while stage-3 is for training multi-page, crop, and other extremely-length max token (e.g., 8K long texts) scenarios.

In summary, the design and data selection for each of our training stages are not aimed at performance, but rather at conserving GPU resources. Resource-saving is essential in training LVLMs, especially when dealing with extremely long texts, such as document OCR. We did not conduct ablation experiments for each stage because, given our computational resources, each stage is necessary, and the absence of any stage would prevent the completion of the training process.

[Presentation] Due to space constraints, we did not include too many sub-sections or architectural diagrams. We are considering reorganizing the structure to enhance understanding of our model.

[Your question]

• In the LVLM field, particularly given the strong memory capabilities of LLMs, models usually choose to train for only one epoch, and our setting aligns with this prior knowledge. Multi-epoch training is only relied upon by traditional computer vision models.
• For samples around 2000 tokens, the resource consumption for training GOT (which requires 20G memory) is much lower compared to Qwen-VL-max (which needs 1600G memory). When deployed under VLLM, GOT achieves an inference speed of approximately 1000 tokens/s on a 3090 GPU, which is several tens of times faster than the 72B Qwen-VL-max.

Sincerely hope that my response has resolved your concerns.