UNIT: Unifying Image and Text Recognition in One Vision Encoder

Thank you for recognizing the contribution of our model to the community. We will make it publicly available in the coming days. We understand that your main concerns and rating stem from some technical details and test settings. Below, we will address these issues one by one. We sincerely hope you can appreciate the significance of our work.

Q1. Unification of image and text recognition

As claimed in our title, we unify image and text recognition into one single encoder (not just "one module" as mentioned in your comment). This unification lies in the shared encoder, not the decoders. The two lightweight decoders primarily serve as auxiliary modules for training and are discarded during actual application.

In our experiment, to evaluate image and text recognition abilities independently, we retain the vision decoder for zero-shot classification and the language decoder for OCR. We recommend users pair our encoder model with a large-scale decoder model (e.g., Vicuna-7B) to enhance both natural image and document-related tasks.

Q2. Some more details

1. Data preparation: We are unsure which specific part you find unclear. We have detailed the data sources, amounts, resolutions, and the ratio of natural images to documents in lines 207-212 and 233-241. I guess your confusion is about generating synthetic datasets of PDF documents, they are rendered with Latex/HTML codes from arXiv papers and websites, as in Nougat. If you have any other questions, please let us know.

2. Effectiveness of random feature sampling:

• Ablation study: In Table 4 of our paper, the results of "w/o feat. sampling" under "Exchanged Resolution", indicate a significant performance drop (31.66% vs. 80.49%) when the sampling strategy is disabled.

• Theoretical explaination: Images have significant spatial redundancy; a missing patch can be recovered from neighboring patches through high-level understanding of parts, objects, and scenes. MAE[1] has demonstrated that even when 75% of image patch features are randomly dropped, the original image can still be reconstructed using a lightweight decoder.

• Advantages: Randomly dropping vectors acts as a regularization technique, reducing noise in gradient calculations. This helps prevent overfitting, improves generalization, and leads to more stable and faster convergence.

  [1] Masked Autoencoders Are Scalable Vision Learners, CVPR2022.

Q3. Scale Robustness

1. Scale robustness: This refers to the model's ability to maintain consistent perceptual capabilities across different input resolutions. Our model achieves scale robustness via inter-scale finetuning. The ablation study is presented in lines 312-316 and the 1st row in Table 4, which demonstrates a significant improvement in "Exchanged Resolution" (0.40% vs. 80.49%).

2. Input scales: Our model can be trained with any resolution below the maximum limit. We chose two primary resolutions for training to ensure consistent tensor sizes, simplifying batch processing and optimizing resource use. UNIT is trained on these two resolutions and tested on different ones. The Table below shows zero-shot classification and OCR results at various resolutions, demonstrating that our model generalizes well and maintains consistent performance.

We also list some experiments in our paper that use different input size:

• In Table 4, we not only evaluate our model on zero-shot classification and OCR on their "Commonly used resolution" (image 224 and document 896), but also show the results on "Exchanged resolution" (image 896 document 224).
• As in line 278, the input resolution is set to 512x512 for semantic segmentation.

3. More experiments: We also evaluated ViTs commonly used in MLLMs on zero-shot classification. These ViTs, pretrained on fixed resolutions, perform poorly as the difference between input size and pretraining resolution increases. In contrast, our model maintains consistent perceptual ability across different resolutions, with no sharp drop in performance.

Q4. Test setting

The two lightweight decoders primarily serve as auxiliary modules for training and are discarded during actual application. In our experiments, they are retained to independently compare image and text recognition abilities against state-of-the-art methods on zero-shot classification and OCR tasks.

UNIT serves as a fundamental vision backbone, providing general visual embeddings for both natural images and documents. We recommend pairing our encoder with a large-scale decoder capable of handling both image and text information. For example, integrating the encoder with a LLM to form a MLLM supports a wide range of applications, including recognizing and understanding standalone images or documents, as well as scenarios where images and text appear together.

Q5. Position embedding

The position embeddings are set based on the model's maximum resolution. For example, if initialized to 64x64xd, where d is the hidden size, this corresponds to a maximum resolution of 896x896 with a patch size of 14. For an image of 896x896, the required position embeddings are 64x64xd, so cropping is unnecessary. However, for an image of 224x224, we need to randomly crop 16x16xd from the position embeddings.

Dear reviewer,

Thank you for your thoughtful comments and for engaging in a detailed discussion.

Q2. About random feature sampling

We would like to clarify that the random feature sampling approach is indeed applied only to natural images, not document images. As mentioned in lines 242-244 of our paper, this technique is specifically designed to enhance training stability and efficiency when computing the token-wise feature reconstruction loss for high-resolution natural images. The purpose of this feature reconstruction loss is to maintain the original image recognition capability, and it is solely utilized for natural images. Therefore, the concern you have about random sampling causing a loss of text information will not occur.

Q5. About position embedding

Our position embedding is 64x64xd, but for input sizes smaller than 896x896, such as 224x224, a 16x16xd position embedding is required. There are generally three possible solutions: cropping, interpolation, and training another 16x16xd position embedding. We chose cropping for the following reasons:

1. Cropping is a more straightforward approach compared to other methods, as it avoids additional computational costs and memory usage.
2. As mentioned in lines 238-239 of our paper, the low-resolution document dataset consists of images cropped from high-resolution documents. Therefore, we use cropped position embeddings for low-resolution inputs to preserve local structural information, ensuring consistent spatial feature representation across resolutions. For natural images, both interpolation and cropping are viable, and we selected cropping for its simplicity.

We hope this explanation addresses your concerns. If our response has clarified the issues and alleviated your concerns, we would sincerely appreciate it if you could kindly consider updating your score.

Thank you for the constructive comments. We understand your main concerns are related to resolution settings and scaling behaviors. We will address these issues one by one and hope you can recognize the significance of our work.

Q1. Training with different resolutions

Our model can be trained with any resolution below the maximum limit. We chose two primary resolutions for training to ensure consistent tensor sizes, simplifying batch processing and optimizing resource use. UNIT is trained on these two resolutions and tested on different ones. The Table below shows zero-shot classification and OCR results at various resolutions, demonstrating that our model generalizes well and maintains consistent performance.

Q2. Different document input size

To ensure efficient data loading and processing during training, we preprocessed the document images by padding them to square shapes. During inference, we maintain the original aspect ratio of the input images.

Q3. Scaling behaviors

1. Model scaling: In our paper, we primarily present the UNIT model with 0.6B parameters. To scale up the model, we added 18 more transformer layers, creating a new version with 1B parameters. After training this UNIT-1B model using the same settings and data, we observed further improvements in almost all the benchmarks.

2. Data scaling:

• More Natural Image Data: We added 5M LAION-COCO data to $\mathcal D^I_{\times 1}$. Increasing the description data of natural images did not lead to significant performance gains, as the original CC3M dataset already covers most common visual concepts in downstream VQA tasks.

• More Document Data: We added 1M document data (from Nougat) to $\mathcal D^T_{\times 4}$, resulting in a slight improvement in document-oriented QA tasks, as the new data provided more diverse samples for different fonts and sizes.

• Finetuning with SFT Data: Beyond freezing the encoder model as done in Llava experiments, we further finetuned the encoder model with the LLM during SFT. This led to significant performance improvements, especially in document-oriented QA tasks, indicating the necessity of high-quality instruction data.

  Setting	VQAv2	GQA	OKVQA	DocQA	ChartQA	InfoVQA
  UNIT(ours)	72.3	60.4	55.5	43.0	46.0	28.7
  More natural image data	72.5	60.5	54.9	42.4	46.3	28.3
  More document data	71.9	59.7	54.3	44.5	47.7	29.8
  Finetune with SFT data	76.7	63.4	56.9	67.6	63.4	36.2

3. Resolution scaling: Scaling up the maximum resolution increases computational costs for training the encoder with higher-resolution images. To address this, we follow existing document MLLMs (mPLUG-DocOwl, Qwen-VL, and Monkey) by using a grid slicing strategy, dividing each image into grids. The encoder generates visual tokens for each grid, which are then concatenated to form the final representation. Despite using LLMs with similar parameters, our method achieves superior performance, as shown in the table:

Q4. Zero-shot cls results in Table 3

This result is reasonable because zero-shot classification evaluates the alignment between the vision and text encoders. When the feature reconstruction loss $\lambda=0$, the vision encoder cannot ensure proper alignment with the text encoder of OpenCLIP-H, leading to the failure of zero-shot classification. CapPa does not use the CLIP models' text encoder for zero-shot classification. Instead, it uses LiT[1], an efficient method to equip any pretrained vision backbone with zero-shot classification, to learn a text embedding that matches the embedding of their pretrained vision encoders.

• [1] LiT: Zero-Shot Transfer with Locked-image text Tuning.

Q5. About LLaVA experiments

The resolution is determined by task requirements, not image type. Low resolution is used for global understanding (e.g., VQA), while high resolution is used for fine-detail perception (e.g., DocQA). The token number is more important and effective than the resolution. Using a 896 resolution for natural images does not improve performance, but increasing the token number to 576 can enhance it.

Q6. Replace Q-former

Q-former reduces the number of visual tokens for images. An image of 896x896 generates 4096 tokens, which is too computationally intensive for a lightweight language decoder and slows convergence. In our training framework, Q-former should be replaced with modules like pooling layers that reduce token count. However, when integrated with a large-scale LLM like Llama-7B, which supports long input sequences, Q-former can be replaced with MLP layers.

After reading the authors' feedback and other reviewers' opinions, I would like to thank the authors for their rebuttal.

The rebuttal addresses most of my concerns, and thus, I raise my score. I would like to note that even though the authors addressed my concerns, I believe that the paper needs some sort of revision, as it appears that many basic concerns were needed to be clarified in the rebuttal. I strongly encourage the authors to fix the main comments mentioned by the reviewers.

Thank you for your comprehensive and encouraging review. We are pleased to hear that you appreciate the technical contributions and the significantly improved performance of our proposed visual encoder, UNIT, along with the intra/inter-scale training paradigm. In the following, we will respond to your concerns and questions:

Q1. Compare with RADIO

The main contribution of this paper is the successful integration of text recognition capabilities into a vision encoder that was pretrained solely on natural images, while still preserving its original image recognition abilities. In our method, the text recognition capability is integrated via a language decoding process, and the image recognition ability is preserved through model distillation. This approach differs from pure model distillation methods like RADIO in two key aspects:

1. Training tasks: Our method employs a multi-task training framework that jointly optimizes distillation, image captioning, and OCR tasks. This necessitates careful model design and implementation to balance these different objectives, making the approach more challenging than optimizing solely with distillation tasks, as done in RADIO.

2. Capability upper bound: In pure distillation methods, the teacher models represent the upper limit of the student's capabilities (e.g., RADIO's performance on zero-shot classification and segmentation is within 3% of that of their teacher models), which may limit the student's learning potential. In contrast, our method learns text recognition from millions of data samples, allowing it to surpass state-of-the-art text recognition models by significant margins, ranging from 2% to 42%.

Q2. Experimental settings about teacher models

The compared models use DINOv2 and OpenCLIP-H as their teachers, while we use RADIO-H, which is also distilled from DINOv2 and OpenCLIP-H. Therefore, both the compared models and our model share the same foundational teacher knowledge derived from DINOv2 and OpenCLIP-H. We believe this makes the comparison reasonable.

Q3. High-resolution document inputs

UNIT supports input images with a maximum size of 896x896. For images with resolutions higher than this, we employ a grid slicing strategy, where each image is divided into several grids according to their aspect ratio. For example, when the maximum grid number is set to 4, the grids could be {2x2, 1x{2,3,4}, {2,3,4}x1}, each of size 896x896. The encoder generates visual tokens for each grid, which are then concatenated as the final representation of the image.

This slicing strategy fragments the original input, disrupting contextual continuity and spatial geometry across patches. However, training the encoder directly with high-resolution images incurs extremely high computational costs. To balance these factors, we should minimize the number of splits. Our method extends the supported resolution range compared to conventional ViTs, with a maximum resolution of 896x896.

Q4. Comparison with existing document MLLM

We create an MLLM using UNIT as the vision encoder. During training, we utilize the CC3M dataset along with 2 million document images for pretraining, and SFT data from LLAVA-665K, including the training sets of DocQA and ChartQA. The document images are divided into grids of size 448x448 to ensure consistency with Monkey[1]. We compare our method with existing document MLLMs, including Monkey[1], Qwen-VL[2], and mPLUG-DocOwl[3], all of which use 7B LLM models. As shown in the table below, our method achieves superior performance on document-oriented VQA tasks.

The main difference between these MLLMs and our method is that their vision encoders are pretrained solely on natural images and are trained with LLMs directly on document QA tasks by adding LoRA [1] or through direct finetuning [2,3]. In contrast, UNIT, as a vision encoder, is pretrained on both natural images and documents with fundamental recognition tasks, i.e., image captioning and OCR. The advantage lies in two aspects: First, pretraining on fundamental recognition tasks makes UNIT more versatile and better suited for downstream document tasks. Second, UNIT's early integration of text recognition capabilities provides a robust initialization for document MLLM training, resulting in more stable training and faster convergence.

To address your concern, we will add these discussions in the revised manuscript.

• [1] Monkey: Image resolution and text label are important things for large multi-modal models, CVPR 2024.
• [2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.
• [3] mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding

Q5. Reference and citation details

Thank you for your valuable suggestions. We will address the issues as follows:

1. The reference for Figure 2 will be added in the appropriate sections, specifically in lines 204 and 232.
2. We acknowledge the incomplete information in the references. The citation for the paper you mentioned, "[6] A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization, 2021," will be updated with the complete details.

We appreciate your attention to detail and will ensure these corrections are made promptly.

Dear Reviewer,

Thank you for your thorough review and constructive feedback throughout the process. We appreciate your recognition of our efforts to address your concerns.

We have carefully considered your suggestions and are committed to revising the paper as you recommended. These revisions will further strengthen our submission, and we are grateful for your insights that have guided us in this direction.

Thank you again for your support!