PaLI-3 Vision Language Models: Smaller, Faster, Stronger

[W1]

We thank the reviewer for the suggestion. We would like to note that we do not claim contrastively pretrained visual encoder is always superior to a classification-pretrained one for all purposes. Instead, we reported our observations on contrastively trained SigLIP that “despite SigLIP slightly underperforming on image classification, it performs much better on Visual Language tasks.” (See our contribution #1) with a focus on the relative advantage of contrastively trained models on VL tasks, which we believe holds true for similarly trained models including OpenCLIP. That said, we agree with the reviewer that the comparison to OpenCLIP may provide a complementary datapoint for the community. We are working on setting up and performing such a comparison and are trying to add an additional baseline based on OpenCLIP before the final version of the draft.

[W2]

We agree with the reviewer that it is important to study the scaling laws to check the effectiveness on a larger scale. The result in Table 1 is about the scaling behavior, where we first tested SigLIP sizes B/16 and L/16, and the promising scaling behavior was exactly what drove us to push SigLIP to 2B (G/14) parameter scale used for PaLI-3.

[W3]

We thank the reviewer for this suggestion. Please see the two main points below. We will expand and add to the revised paper.

• PaLI-3 used SigLIP pretrained ViT, while previous versions used the classification pretrained ones.
• The training mixture of PaLI-3 is based on the learnings of PaLI-X, with a few key new components, including the further filtered and refined WebLI dataset, especially on the OCR data, and we curated and added web-scale document images in the training.

[Q1] Code release

We are actively working on a potential open-sourcing of our model, however there are still aspects related to the open-sourcing effort that we are still figuring out.

[Q2] Using SigLIP for much larger LLMs

We thank the reviewer for the suggestion. Indeed, we are also very interested in the potential of SigLIP in large LLMs. We plan to explore this in the followup work with the potentially further scaled-up SigLIP variants.

[Q3] Difference between PaLI-3 and previous versions. Similar to W3

We thank the reviewer for this suggestion. Please see what we listed above, and we will expand and add to the revised paper

[Regarding the first point from the reviewer]

It might sound intuitive to the reviewer, but to date, there was no direct comparison published, and by looking at individual papers in isolation, it may have seemed that JFT-pretraining leads to superior combined models, given PaLI (arXiv:2209.06794) and PaLI-X (arXiv:2305.18565) results.

In this work we actually perform apples-to-apples comparison using contrastively trained SigLIP ViTs and discriminatively trained ViTs with the same architecture and and quantify the wins and losses. Our comprehensive experiments show that there is not a simple winner: while the contrastively pretrained SigLIP models still lose to the discriminatively trained counterparts on visual classification tasks, SigLIP-based PaLI-3 shows superior performance on Vision-Language tasks. We believe that our experiments provide concrete data points and insights to the community beyond just intuitions.

[Regarding the second point from the reviewer]

In this work, we made substantial improvements over the PaLI training recipe. Our contributions to the training mixture is generally applicable. Even in the reviewer’s own hypothetical statements, “Actually incorporating these improvements could probably benefit any other model compared with the proposed one.”, the reviewer acknowledged the significance of our training recipe.

It was very surprising to us as authors that such a compact model (at only 5B) was able to outperform a related but much larger-capacity model (PaLI-X at 55B) across so many tasks. One of the goals of this paper was to highlight what parts would be responsible for such a surprising result, and we consider that the research community would be equally interested in understanding it.

[W1]

We thank the reviewer for the suggestion. We would like to note that we do not claim that “contrastively pretrained visual encoder is superior to a classification-pretrained one for all purposes”. Instead, we reported our observations on contrastively trained SigLIP that “despite SigLIP slightly underperforming on image classification, it performs much better on Visual Language tasks.” (See our contribution #1) with a focus on the relative advantage of contrastively trained models on VL tasks, which we believe holds true for similarly trained models including OpenCLIP.

For OpenCLIP, We have not tried it at the time of submission, since SigLIP is also open-source and significantly better across benchmarks. However, we agree this can be an interesting ablation, for which we have started to set up and are trying to add an additional baseline based on OpenCLIP before the final version of the draft.

[W2]

We expect the zero-shot performance to be relatively poor, since there is no task aimed for this in the pre-training mixture, and hence the model cannot know "the coco vocabulary" from which it should generate. PaLI-3 is also not a chatbot, but rather a (compact) foundation model capable to be transferred to various tasks.

Meanwhile, few-shot in-context ability is a good idea that we could and likely should add in the pre-training mixture, and would provide an in-between alternative between fine-tuning and zero-shot.

[W3]

Thanks for the question, it highlights an unclarity in the write-up. The stage0 pre-training does not use a UL2 encoder-decoder. It is trained with a transformer encoder for the text with a similar size to ViT-L, following SigLIP (and like CLIP), which we then discard in the following stages. Thus, no such consistency in the language model is required. We will emphasize this aspect better in the paper’s next version.