ColPali: Efficient Document Retrieval with Vision Language Models

We thank the reviewer for their feedback, which highlights our contributions in terms of benchmarking and methods along the concept of vision-based document retrieval first introduced in this work.

(W1) Benchmark Samples: Figure 3 (right) and Figure 6 illustrate two varied query-page pairs present in ViDoRe, and an additional document example is given in Figure 5. Furthermore, the modalities, thematics, and languages of our benchmark are detailed in Table 1 and provide a comprehensive understanding of the benchmark's scope. Note that the ViDoRe benchmark can be easily visualized using the HuggingFace dataset viewer, which we will link to in the final manuscript. Encouraged by the reviewer, we have added additional query-page illustrations in the appendix.

(W2) Retrieval Candidate Space Clarification: The retrieval candidate space for each ViDoRe subtask is explicitly defined in Table 1, detailing the language, number of queries, number of candidate documents and giving a brief modality and thematic overview of each of the subtasks.

(W3) ColPali Model Details: As indicated throughout the paper, ColPali produces multiple vector embeddings for each page and for each queries. As indicated in the reviewer’s summary, we leverage the Late Interaction operation to match the (N_q) multiple query vectors to the (N_d) document vectors to produce a scalar matching score. This operation is detailed in Sec 4.1 L284, illustrated in Figure 1, and mentioned throughout the paper. We believe clearing up this potential misunderstanding immediately explains all other remarks under this bullet point, as it should thus appear more clearly how document representations are generated (L280-283, Fig. 1, L378), to what correspond N_q and N_d (L284-288), and the fact neither <bos>, <eos>, nor pooling is necessary since all token embeddings are used. Pooling for the single-vector BiPali is detailed L363. We have clarified the notations (L286) and further re-explicited that all token embeddings are kept (L280). We note that our codebase is made fully available, and has since the paper’s release led to multiple successful replication attempts.

(W4) Broader Evaluation: The ViDoRe benchmark is already quite exhaustive, and it does in fact already include multiple popular publicly available datasets (DocVQA, InfoVQA, TAT-DQA, ArxivQA) (L194), in addition to several other custom tasks spanning various languages and modalities. The reviewer suggests using datasets from the VLM2Vec paper but they are partially redundant with ViDoRe - some of the most relevant subtasks are already included in our benchmark (InfoVQA, DocVQA) -, and the preprint was released one week after the ICLR deadline. As for MP-DocVQA, this dataset is an extension of DocVQA in which a few additional context pages are added to the correct gold page. This dataset is originally designed for multi-page VQA, and since we have repurposed DocVQA for retrieval and already consider 500 pages as potential candidates for each query, we believe this task to be redundant, hence why it is not included in the current version of ViDoRe. We believe the 10 current ViDoRe subtasks that cover multiple languages (French, English), modalities (pure text, tables, figures, infographics, mixed-modality documents) and topics, are already a fairly diverse set of experiments and highly correlate with model performance. We acknowledge the VLM2Vec benchmarking efforts are complementary to the research direction introduced in this paper and have added a mention in the future work section.

Questions

Page-Level Retrieval Capability: As made clear thoughout the paper, yes, ColPali retrieves at a page-level granularity. DUDE is out of the scope of this work, as it is intended for multi-page QA and features multi-hop questions that necessitate multiple pages to be answered, as well as questions with no answers within the corpus. It is also largely similar to DocVQA which is already included in our benchmark. MP-DocVQA is mentioned in (W4).

Text Retrieval Functionality: ColPali generates embeddings for both text (using only a LLM) and images (by feeding soft tokens produced by a vision model to the LLM) in a shared embedding space. As a result, beyond text-to-image retrieval, it is natively capable of performing text-to-text retrieval (similar to ColBERT), as well as image to image matching. Although we are aware of successful efforts in this direction, benchmarking this is beyond the scope of this paper which focuses on text-to-document image retrieval, which we believe has the potential to truly transform current document RAG practices This is being confirmed by the significant enthusiasm we have observed from both the industrial and academic community since the release of our preprint.

We hope that under the light of our clarifications, the reviewer will reconsider their assessment of our paper and their score, and thank them once again for their review.

We thank the reviewer for their rebuttal and appreciation of our work. In line with the reviewer's intuition, we can confirm that the open nature of our work and resources have been capital in promoting the development of our introduced concept of vision-based document retrieval since the release of our preprint.

Generality and Out-Of-Domain Application Scenarios.

We fully agree with the reviewer's assessment that validating the generalization capabilities of our method is of interest. In our paper, we have shown this in two ways:

• Out-of-distribution tasks: Several tasks in the benchmark are by design excluded and out-of-distribution w.r.t. to the training set. Typically, TabFQuAD and Shift Project are two tasks which are in the French language (all training data is in English), and covering modalities and topics not explicitly present in the training set (industrial tables, ecological topics). ColPali remains by far the best model on these domains, even though some baselines (Unstructured + BGE-M3) do cover French in their training set. This is a first step in showcasing the general nature of our technique.

• Extra Experiments: To further confirm the insights detailed in (1) about the generalization capabilities of our method, we studied out-of-domain generalization (L502) through an extra experiment, in which we replace our (partially in-domain) training dataset by an entirely disjoint dataset comprised of synthetic data which is fully OOD w.r.t. the ViDoRe benchmark. As stated in the paper, “results on ViDoRe show the performance drop is minor (−2.2 nDCG@5), still outperforming the closest baseline method by over 12 points. These results showcase ColPali generalizes well outside of its training distribution.”

As benchmark results are often only part of the story, we have also released public demos hosted on HuggingFace spaces enabling users to test out ColPali for their given use cases and have had overwhelmingly positive feedback. Several preprints released since our preprint release have extensively and independently evaluated our ColPali method and have further demonstrated the very generalist nature of our model on previously unseen data distributions.

We hope this rebuttal has reassured the reviewer regarding the perceived potential weaknesses of our work!

We thank the reviewer for their comments that highlight the importance of our work and the effectiveness of our methods.

Novelty

We believe our work to be a true paradigm shift for RAG systems. Since the release of our preprint, we have observed a huge interest in both the academic and industrial community in the “document retrieval from images” concept we introduce in this work. Although we do leverage pretrained models (PaliGemma) as building blocks, and existing multi-vector matching techniques (ColBERT), our work is the first to consider constructing document embeddings from images with vision models, and to have showcased the feasibility and many benefits of this approach. In the paper, we show this is enabled by several techniques we introduce and successfully apply to this document retrieval use case for the first time, namely

• Repurposing generative vision-language models that align image token embeddings to the text token latent space and enable understanding text semantics “from image pixels”
• Largely boosting performance by leveraging cross-modal multi-vector embeddings that better encode rich visual information
• Introducing and releasing new synthetic datasets for this purpose and releasing all project artifacts

Our released models enable huge boosts for many existing use cases–along both the retrieval performance and the indexing speed axes–, enable many novel future explorations, and have truly demonstrated a promising alternative to the previous document RAG standards prior to our preprint.

Latency Details

The reviewer indicates the latency results are poorly detailed in the paper. In the submission, quantitative latency results for image embeddings are given in Section 3.2 and illustrated in Figure 2. As indicated in Figure 2’s caption, appendix Section B.4 further complements these results by offering expanded details on how these latencies were measured. To address the reviewer’s concerns, we have moved some details in the main paper and expanded the appendix with additional details. Query latencies are also detailed in section 5.2 (L399). Modifications are made in red.

We thank the reviewer once again for their suggestions, and hope this rebuttal and the induced modifications have cleared some interrogations and improved the quality of our work. If this were to be the case, we kindly ask whether the reviewer would consider improving their review score.

We thank the reviewer for their detailed review emphasizing the interest of our method and the value of our benchmark.

Clarifying the construction process

This work is the first to consider doing document retrieval purely from images, rather than extracted text content, marking a significant paradigm shift from previous techniques. Through the construction ablations in Section 5.1, we show that this breakthrough is possible through the combination of (1) a carefully crafted dataset that includes synthetic data, (2) pairing a pre-trained language model to the vision model to be able to capture text semantics from the image, and (3) using multi-vector embeddings rather than a single vector to better capture the vast amount of information present in a visual document. To emphasize this point, we have added an introductory sentence to Section 5.1 and slightly clarified the structure, adding some details for readers less aware of the technical details of the underlying models. We have also added a model glossary in Appendix E further detailing SigLIP, PaliGemma and ColBERT for interested readers. Note that replication is also (and has been) easily enabled by the availability of the training code.

In-Distribution Training

The reviewer correctly notes that some tasks of the ViDoRe benchmark are in-distribution with respect to the training set. While this is common practice in embedding papers (MTEB evaluation), we were also interested in estimating the impact of in-domain training w.r.t. methodological innovations. This is done in three ways:

• Out-of-distribution tasks: Several tasks in the benchmark are by design excluded and out-of-distribution w.r.t. to the training set. Typically, TabFQuAD and Shift Project are two tasks which are in the French language (all training data is in English), and covering modalities and topics not explicitly present in the training set (industrial tables, ecological topics). ColPali remains by far the best model on these domains, even though some baselines (Unstructured + BGE-M3) do cover French in their training set. This is a first step in showcasing the general nature of our technique.

• Finetuned baselines: While being unable to finetune Unstructured-based methods because of the prohibitive cost of extracting text for the 40000+ pages of the training set with the same settings as what we run at inference (using layout detection, etc…), we do finetune the best vision-based baseline, the SigLIP model, on the same training data. This results in the BiSigLIP model, which presents performances that are vastly inferior to ColPali across the board (-22.7 nDCG@5). This shows the interest of our method goes beyond a strong dataset.

• Extra Experiments: To further confirm the insights detailed in (1) and (2) about the generalization capabilities of our method, we studied out-of-domain generalization (L502) through an extra experiment, in which we replace our (partially in-domain) training dataset by an entirely disjoint dataset comprised of synthetic data (DocMatix) which is fully OOD w.r.t. the ViDoRe benchmark. As stated in the paper, “results on ViDoRe show the performance drop is minor (−2.2 nDCG@5), still outperforming the closest baseline method by over 12 points. These results showcase ColPali generalizes well outside of its training distribution and demonstrate that our results are not unreasonably boosted with respect to baselines (BGE-M3) that cannot be finetuned on the same data.”

We are thus very confident that the results reported are not due to in-domain training and that our method is very generalizable to OOD data. Furthermore, as we have released our training data, future work can leverage it (and already have) to disambiguate methodological innovations from data innovations.

We hope this rebuttal has reassured the reviewer regarding the perceived weaknesses of our work, and hopefully incited them to consider improving their grade.