OmniContrast: Vision-Language-Interleaved Contrast from Pixels All at once

Thank you for your detailed observations and questions. Below, we provide clarifications and propose adjustments to enhance the manuscript based on your valuable feedback.

Contribution:

We sincerely request the reviewer to re-evaluate our contribution from two perspectives:

• New Training Fashion: As highlighted by Reviewer #naQ8, we are the first to explore the potential of image-text interleaving documents for training CLIP-style models, which has been underscored in the vision-language research community.
• More Unified Model: Our approach is capable of directly handling long text and images with embedded text in a unified model, as evidenced by Table 5 and Table 3—capabilities that CLIPPO or other baseline models lack. In Section 6, we showcase that OmniContrast learns a more unified omni-modality representation, which indicates unifying in pixel space can further reduce the modality discrepancy.

Data Augmentation:

We understand the importance of clearly explaining our data augmentation strategies to address concerns about training sample quality. Our augmentation techniques include modality masking and text masking, designed to enhance diversity and robustness during training.

• Modality masking involves randomly dropping one modality from image-text interleaved samples, such as removing the text while retaining only the image.
• Text masking randomly removes sentences from the beginning or end of the text content when the text contains more than four sentences and exceeds 250 characters.

Detailed descriptions of these strategies are provided in the supplementary material. The effectiveness of these augmentations is demonstrated through ablation studies in Section 6.1, which show that they not only increase training diversity but also improve the model’s robustness to unseen inputs. Additionally, we include visualizations of our training data in the supplementary material to further illustrate these techniques.

Figure2 & Terminologies:

Thanks for the suggestion!

• We will improve the readability of Figure 2.
• We will revise the manuscript to clarify that ``omni-modality" refers to an image, text, and image-text interleaving. This term will be explained early in the paper to avoid confusion.
• We will add a footnote clarifying that here it refers to visually presenting text, akin to copying and pasting onto an image, to avoid confusion.

Converge and Benefits from Omni-style:

Based on the training loss curves, we observed the training in omni-style makes it harder to converge than in image-text style. However, our experiment results, e.g. Table 1, have shown that training a vision encoder with omni-modality data generally helps each modality learn better.

Limited data:

The training data in OmniContrast is imbalanced across different modalities. Specifically, the proportion of image-to-image pairs is smaller than that of text-to-text pairs due to the limited number of images compared to the larger volume of text chunks. Consequently, as shown in Table 2, the performance on the image-to-image task is lower than that of the text-to-text counterpart.

About third-party baselines: It is not true.

We do include third-party baselines in Table 4 (M-BEIR) and Table 5 (MTEB). There are several baselines included from third parties, such as SigLIP (Table 4), BLIP (Table 4), BLIP2 (Table 4), Glove (Table 5), Komninos (Table 5), BERT (Table 5), and SimCSE-BERT-unsup (Table 5).

Common Benchmarks & Q2: It is also not true.

Our evaluations have included two common benchmarks: M-BEIR (Table 4) and MTEB (Table 5). The comparison is presented in Line 324-Line 354. OmniContrast focuses on image-text interleaving and long-text scenarios. While these traditional image-text retrieval benchmarks and GLUE typically are short sentences and are well-explored. Therefore, to better evaluate the capability of image-text interleaving and long-text understanding, we chose the M-BEIR (more diverse text-image retrieval settings) and MTEB (more long text settings). Moreover, the image-text retrieval setting and some tasks from GLUE are included in these well-developed benchmarks. We believe these benchmarks collectively offer a more comprehensive evaluation. We kindly encourage the reviewer to refer to these results to re-evaluate our work.

About Section 5.2:

Yes, in section 5.2 we only use the vision encoder of CLIP / OpenCLIP for fair evaluation. In Section 6.1 we provide the result of the full CLIP / OpenCLIp model for further discussion.

Discussions about Multi-Modal Large Language Models:

Thank you for the suggestion. Multi-modal Large Language Models (MLLMs) are not specifically tailored for retrieval tasks. Their primary strength lies in handling interleaved inputs for generative tasks, such as image captioning and question answering with both visual and textual inputs. For example, in Table 4 M-BEIR benchmark, BLIP2[1] is similar to Emu[2] powered by the large language model and shows very limited performance on various retrieval settings. Applying MLLMs to retrieval tasks [3] is another promising research problem, but is not our focus. We will discuss these methods in Section 6 of the revised paper to address this perspective.

[1] Li, Junnan, et al. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML 2023

[2] Sun, Quan, et al. Emu: Generative Pretraining in Multimodality. ICLR 2023

[3] BehnamGhader, Parishad, et al. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. COLM 2024

Thanks for the constructive feedback! We hope to clarify the confusion and answer the questions below.

Comparison Baselines:

Our research emphasizes the importance of image-text interleaving and long-text scenarios. While traditional benchmarks like VQAv2 and GLUE tend to focus on short sentences and have become somewhat outdated. Therefore, we have selected two more comprehensive benchmarks: M-BEIR, which offers more diverse text-image retrieval settings, and MTEB, which provides more extensive language understanding scenarios with longer text. Additionally, the more complex QA setting and some tasks from GLUE are included in these well-developed benchmarks (Line 324-354, Section 5.3 and 5.4). By integrating these diverse benchmarks, we offer a more comprehensive evaluation result. We respectfully invite the reviewer to carefully examine these results to gain a deeper understanding of our work.

Necessity and Effectiveness of Unifying:

• Necessity: We would like to emphasize that unifying everything into pixels can avoid specialized design for diverse modalities, which significantly reduces the complexity of the training and inference pipeline. Moreover, our approach is capable of directly handling images with embedded text and long text, as evidenced by Table 3 and Table 5—capabilities that baseline models lack.
• Effectiveness: Extensive experiments in Table 6 show that OmniContrast surpasses the separate encoder in terms of smaller training data scales and model sizes. We showcase that OmniContrast learns a more unified omni-modality representation, which indicates unifying in pixel space can further reduce the modality discrepancy. In Section 6, we provide a detailed discussion of the necessity and effectiveness of unifying pixels.

Comparison on image-text paired data:

In our experiments, we have included an image-text data baseline trained on the LAION 40M subset for comparison, i.e., Im-Tx baseline trained on LAION-40M (L276 in Table 1, L289 in Table 2&3, Table 4 Im-Tx$_{la}$, L349 Im-Tx (LAION) in Table 5). Several notable differences emerged between our approach and the image-text paired baseline.

• For instance, as shown in Table 5, the image-text baseline performs better on the NIGHTS [1] retrieval dataset, where the task requires retrieving identical images. This suggests that the image-text paired baseline is more effective at capturing fine-grained image details.
• However, in terms of text embedding quality, as presented in Table 6, the image-text paired baseline falls significantly behind Omni-Contrast, indicating that our approach excels in text representation as the model is trained with longer text.

[1] Fu, Stephanie, et al. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv:2306.09344.

Thank you for taking the time and effort to review our work.

What Makes Single Modality Good?

• Firstly, we would like to highlight that unifying in a single modality significantly simplifies the model architecture by reducing the complexity of the text encoder and the need for a fusion strategy typically required in separate encoder setups as recognized by the Reviewer #Bmsk.
• Secondly, our model is inherently designed to handle images with image-text interleaving content directly. In contrast, separate encoder models require additional steps for text extraction to fully utilize both modalities. The interleaved data commonly appears in everyday scenarios such as documents, TV shopping broadcasts, advertisements, and more, where extracting text can often be challenging.

Application Scenarios:

• It is important to note that in the case of HTML, image-only or text-only inputs are simply special cases handled seamlessly by our model! For example, as shown in Figure 2, our model already supports such single modality input during training.
• Moreover, as suggested in [1] handling HTML is much more complicated than plain text and requires a much larger input context while it is very straightforward and simple by handling them in image space using screenshots.
• Besides, many other scenarios, such as screenshots [2], slides [3] (as demonstrated in our CSR benchmark), PDFs [4], and scene text [5], present image-text interleaved content in a primarily visual format. These cases pose unique challenges for extracting text content due to their reliance on visual input. Therefore, a unified approach to processing image-text interleaved data is particularly valuable.

The Redundancy of Single Modality:

We acknowledge the additional effort required to re-organize image-text input into the form of screenshots for OmniContrast. However, this process requires a much lower computational cost compared to forwarding text inputs through an additional text encoder. Regarding the resolution, our experiments demonstrate that a carefully chosen resolution strikes a good balance between performance and input quality. For instance, our approach supports a maximum text input length of 1,100 characters (around 275 tokens), while the text input of CLIP is limited to 77 tokens.

Simple Alternatives: We respectfully disagree that these simple alternatives are very effective.

• In terms of the model scale, CLIP-V+T (ViT-L) and UniIR are in the size of ViT-L while our OmniContrast is in the size of ViT-B and their performance is still limited, e.g., 42.81 (ours) v.s 30.72 (CLIP-V+T ViT-L) v.s. 29.31 (UniIR) in overall performance. In the size of ViT-B, our model outperforms the CLIP-V+T (ViT-B) by a large margin, e.g. 42.81 (ours) v.s 25.79 (CLIP-V+T) in overall performance.
• In terms of training data, our training data MMC4-core maintains a relatively small size, i.e., 5M, for which we believe OmniContrast has great potential to be an effective solution when scaling up the model and training data.

[1] Gur, Izzeddin., et al. Understanding HTML with Large Language Models. EMNLP 2023

[2] Chen, Xingyu., et al. WebSRC: A Dataset for Web-Based Structural Reading Comprehension. EMNLP 2021

[3] Tito, R., et al. Document Collection Visual Question Answering. ICDAR 2021.

[4] Araujo, André, et al. Large-scale query-by-image video retrieval using bloom filters. arXiv:1604.07939

[5] R, Ganz, et al. Towards Models that Can See and Read. ICCV 2023