No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

W1: unify table descriptions

Thank you for pointing out this discrepancy, we apologise for the oversight and have fixed this in the paper.

W2: connections to existing work

Thank you for this question. We have included a brief related work section in the main paper in sec 7, describing the differences of our work to the other related work in literature. Further, we discussed our paper’s position in the literature in the last paragraph of the introduction under, "Situating our Contributions in Broader Literature." Lastly, we note that we have included a larger related works section in appx E. We hope this addresses the concern.

W3: qualitative visualizations/theoretical analysis lacking

• Regarding qualitative visualizations: While we share T2I generations for the long-tailed "Let It Wag!" concepts in figs 7, 24-27 as well as misalignment in fig 23, we are happy to provide additional figures visualizing concepts according to frequency in the revised paper.
• Regarding relevant theoretical analysis: While we mention that theoretical analysis is lacking explicitly in our limitations (see appx N), our work is an example of empirical theory [1,2], where we attempt to quantitatively describe real-world behavior, thereby yielding conjectures, which, if proven, yields theory that holds in practice. Such approaches are standard in physics, and has yielded characterizations that have become prescriptive for practitioners (Hoffmann et al). In our case, we have shown that typical power law (log-log) relationshisp between training set size and performance do not hold when downstream performance is related to concept frequency within training sets, where we instead observe log-linear trends, thereby characterizing the detrimental effects of the long tail. We will expand upon this in the revised version of the paper.

Q1: conflicting with works that improve multimodal model generalization?

Thank you for this insightful question. We believe our conclusions should hold regardless of the model architecture and the training objective for any VLM. However, to test this, we investigated two methods that have been empirically shown to improve generalization capabilities of CLIP models: CyCLIP and SLIP. We use 4 different models, each trained with either CyCLIP/SLIP on three different datasets---we then plot our main log-linear scaling results similar to figs 2, 3 for CyCLIP and SLIP models—these plots are in the uploaded rebuttal PDF at the bottom. We observe for both SLIP and CyCLIP models, the log-linear scaling trends hold strong, with high Pearson correlation coefficients, further signifying the robustness of our main results. Hence, we emphasize that our main conclusions hold true even when considering multimodal models that explicitly introduce new training objectives with the aim of improving model generalization.

Q2: insufficient explanation of figs 24-27

We apoloigise for the lack of clarity. The motivation behind figs 24-27 was to provide qualitative insights into failure of T2I models on long-tailed concepts, providing a more detailed analysis of fig 7 of the paper. We used three T2I models to generate images from prompts generated for each concept by Gemini/GPT4. For a more holistic understanding of model failures on long-tailed concepts, we group concepts into a broader semantic category: aircraft (fig 24), activity (fig 25), animal (fig 26) and misc (fig 27), to showcase the broad spectrum of concepts captured and how T2I models fail in each case. This has been detailed in sec 6 of main paper and appx. J.2 but we will expand the section further with clarifying details.

Q3: motivation and implications of work?

Thanks for this question. While we provide motivation in our introduction and have also included broader impacts section (appx O) to highlight the implications of our work for future research in multimodal models, we will make sure to clarify these sections further.

• Motivation: While multimodal models demonstrate impressive "zero-shot" performance, knowing whether this is a result of an overlap between downstream tasks and the pretraining dataset not only calibrates expectations for how these models will perform in-the-wild, but also clarifies if the aforementioned performance could be achieved more efficiently by leveraging improved dataset priors, ie downstream task-aware curation.
• Implications: Given the observed sample inefficient log-linear scaling trend suggests that models indeed are reliant on downstream task concepts being exponentially frequent in their pretraining dataset, this implies that reliable performance when encountered with rare concepts remains an outstanding question for current multimodal foundation models. Based on this analysis, we contribute the "Let-It-Wag!" testbed by aggregating rare concepts, to further future research in this direction.

Q4: tasks beyond classification/retrieval

Thank you for the question—we agree, it is important to consider tasks outside classification/retrieval. Hence, we have studied the performance of 24 T2I models on the task of text-based image-generation and observed the same log-linear performance trend as that of classification and retrieval. Please refer to fig 3 (main paper) and fig 11-15 (appendix) for results of the log-linear claim for all metrics across all 24 text-to-image models. Additionally, we provide a human evaluation on the text-to-image results as well, please see appendix C and fig 16, 17. Finally, we also highlight the difference in performance between head and tail concepts using a quantitative nearest-neighbour retrieval experiment—please refer appendix J. We hope this sufficiently addresses the reviewer’s concern.

L1: mitigating solutions?

Thank you for this insightful question. While our paper does not propose specific solutions, we believe its primary contribution is in thoroughly highlighting the issues with current pretraining strategies for multimodal models across various datasets, pretraining methods, architectures, training objectives, and tasks. Additionally, by releasing the "Let it Wag!" testbed, we provide a straightforward test set for future research to build upon, aiming to improve the generalization of multimodal models to long-tail scenarios. However, we suggest a few potential methods that could be explored to enhance multimodal long-tail:

• Retrieval Augmentation: Enhancing generalization to long-tail concepts can be achieved by utilizing the "world-knowledge" of LLMs to provide detailed descriptions for these concepts. This approach transforms the task from simply recognizing long-tail concepts by name to recognizing them by both names and descriptions.
• Curriculum Learning: Our tested models used random IID sampling during training. However, research into better sequencing of data samples could potentially improve model generalization to long-tail concepts by inducing more transferable feature representations in VLMs.
• Synthetic Data: Addressing the issue of long-tail concepts in web-sourced datasets may not be feasible by merely increasing data samples. There will likely always be low-data density regions in the pretraining data distribution. Using synthetic data, either through procedurally generated samples or text-to-image models, could be a viable mitigation strategy.

We hope these suggestions provide valuable directions for future research and contribute to the development of multimodal models capable of better generalization. We will add these points to the future works section in the appendix.

L2: stronger focus on generation and Let-It-Wag! dataset

We thank the reviewer for raising this point. We would like to respectfully argue that we believe our image-text contributions are equally as important to the other contributions, please see response to W2. Re. focus on Let-It-Wag! dataset, we agree that it would be prudent to emphasize its construction/properties further. We will make these points clearer in the paper and hope this clarifies the issue.

L3: usefulness of insights gained from CC/YFCC experiments?

We thank the reviewer for raising this important point.

• CC/YFCC datasets used for important VLM pretraining methods. While we agree that smaller datasets like CC-3M/YFCC-15M might not yield insights which are practically relevant for SoTA performance, we still believe that it is an important validation to perform these experiments at that scale. We note that lots of high impact work on CLIP-like models have been empirically validated by pretraining on datasets like CC-3M/CC-12M/YFCC-15M, including CyCLIP, DeCLIP, OpenCLIP, SLIP, FILIP, MERU and LaCLIP.

• Expansive insights spanning a range of dataset scales. Further, we note that each of these datasets from small-scale CC-3M to the large scale LAION-400M employ different data curation strategies, and by showcasing that (1) all datasets have very similar pretraining concept distributions, and (2) models trained on any of these datasets showcase the same consistent log-linear scaling trends, we have empirically showcased the robustness of our main exponential data inefficiency result, and uncovered several important properties of VLM pretraining datasets.

• Significant compute resources. Finally, we would like to point out the significant compute resources it would take to conduct our analysis on even larger-scale datasets like DataComp-1B or LAION-2B—in appx L tab 8, we report the storage and compute costs required for conducting all our experiments—at the LAION-400M scale, we would need almost 10TB of disk space and 2000GPU / 7000CPU hours (this is roughly in the order-of-magnitude required for pretraining CLIP models themselves, see Cherti et al tab 18).

• Testing on other web-scale datasets with Let-It-Wag! Given our results’ robustness across several scales, we have strong reasons to believe that our result will continue to generalize even for larger-scale datasets and models. One point of evidence to further bolster this is that we tested models trained on larger-scale datasets including DataComp-1B, DFN-1B and WebLI-10B, in our analysis on the Let-It-Wag! Dataset in fig 6. We however do not see any significant deviations for models trained on these datasets that would lead us to believe that they showcase different characteristics to the datasets we analysed in our work.

W3: info on Let-It-Wag! construction

Thank you for pointing out this concern, we apologise for the lack of details regarding the construction of the Let-It-Wag! Benchmark. We provide some details here and have also added this information to the appx.

For curating the set of 290 concepts, we collate our frequency estimates from the LAION-400M dataset for all the 4,029 concepts we consider. We then remove all the concepts that have 0 counts to ensure that our final dataset consists of concepts that have been detected atleast once in LAION-400M, this method has also been used in [3] to ensure robustness to noise in the estimation process. For each extracted concept, we then apply the image download and manual cleaning procedure described in appx G.2. Finally, we removed those concepts that had a low number of total images left post the cleaning process, and retained all the other concepts. We then sorted these concepts in ascending order of their estimated concept frequencies, and retained the top 290. These concepts are used as the final list of classes for our Let-It-Wag! benchmark.

W4: high-level insights about long-tail concepts

Thank you for this question. The broad categories of the most long-tailed concepts with a few examples for each are as follows (a majority of them have been highlighted already in Figs 24-27 in the appendix):

• Birds: Western_Scrub_Jay, Cassins_Finch, Prairie_Warbler, Red_eyed_Vireo, Veery
• Animals: flatworm, Tibetan_Mastiff, Scottish_Terrier, vine_snake, newt
• Aircrafts: A300B4, A310, Falcon_900, DHC-8-300, MD-11
• Objects: guillotine, letter_opener, ladle, dust_jacket
• Plants and fungi: mexican_aster, gyromitra, great_masterwort, thorn_apple, cape_flower
• Misc: consomme, stratified_texture, eggnog

Additionally, we will release the full list of these concepts as the classes of the Let-It-Wag! benchmark in the appx.

Q1: stats about Let-It-Wag?

Thank you for raising this question. We provide the required stats below.

• Most frequent concepts: partridge (count=9489), Bank Swallow (count=9489), eel (7907)
• Least frequent concepts: Red-necked Grebe (count=0), SR-20 aircraft (count=0), Globe-flower (count=0)
• Median frequency of concepts: 97.5
• Mean frequency of concepts: 1096.2

We also show the full histogram of concept frequencies for the 290 concepts in the Let-It-Wag! dataset in the uploaded rebuttal pdf. From the histogram, it is evident that most of the concepts in Let-It-Wag! have frequency less than 2000. About half of the concepts in Let-It-Wag! (~140) have a frequency less than 1000. Hence, this histogram sufficiently establishes that our Let-It-Wag! dataset truly captures the long tail.

Q2: variance in fig 2 points

Thank you for the insightful question. We provide zero-shot classification plots for CC-3M, CC-12M, and LAION-400M in the uploaded rebuttal PDF, including 95% confidence intervals for each point. This approach follows the standard practice from works like [1, 2]. Our plots show that the spread at higher frequencies is significantly larger than at moderate frequencies, corroborating the finding that higher frequency concepts are more ambiguous and polysemous. These results support the observed dips in accuracy at high-frequency points. We will include these plots in the appendix and discuss this in the paper.

W1: Dips in accuracy and consistency of log-linear trends?

We thank the reviewer for raising this concern.

• Analysis of drops in high freq concepts. We provide some intuitions on why there are some drops in the trend at high freqs for CC-3M and CC-12M, we investigated which concepts occur at such high freqs, specifically above 10^4. The complete list of the relevant concepts and the corresponding downstream dataset will be added to the paper. From our analysis, we hypothesize two key reasons for a performance dip:

  • Concept ambiguity: we observe many concepts that are homonyms / polysemous (same spelling but different meaning ie can represent multiple concepts at once). Some examples are watch, bear, house, fly, bridge, cloud, park, face, bar, tower, wave, etc.
  • Broad concepts: A concept with a broader scope of definition supersedes a narrower one (concept ‘dog’ vs the specific breeds of dogs seen in imagenet-r ('yorkshire terrier', 'boston terrier', 'scottish terrier', 'golden retriever', etc)). These concepts are too coarse-grained and hence can be visually represented by a diverse set of images. Performance variance of these concepts can be quite high based on the specific set of images given for testing.

  These ambiguities become more prevalent the more ubiquitous a concept is, which is directly tied to its freq obtained from pretraining datasets. Some more examples for a deeper understanding of the diversity of concepts are: cucumber', 'mushroom', 'Granny Smith', 'camera', 'chair', 'cup', 'laptop','hammer', 'jeep', 'lab coat', 'lipstick', 'american-flag', 'bear', 'cake', 'diamond-ring', etc.

• Statistical validity across broad evaluations. Beyond our analysis, while we agree that there are some slight deviations from the log-linear scaling trend, particularly at high frequencies for the CC-3M and CC-12M datasets for classification, we would like to point out that these log-linear trends in general hold strongly across two different tasks and 4 different datasets. We have also validated that these trends are statistically significant by including the Pearson correlation coefficients and conducting a two-tailed t-test to increase confidence in our results.

• Noise in web-scale datasets and frequency estimation. Further, we would like to highlight that web-scale datasets are inherently very noisy, and coupled with that our frequency estimation procedures could be quite noisy due to the inherent scale of our search and estimation procedures. Despite this noise, we observe strong log-linear scaling trends across the board. Further, we point the reviewer to the results in fig 22 of appx where we showcase that the log-linear scaling trends hold even when our frequency estimation pipeline is explicitly made noisier (by varying threshold of RAM++), further signifying the robustness.

We will further ensure that each of the figure captions reflect the exact nature of the consistency of the log-linear trends in the main paper. We hope to have adequately addressed the reviewer’s concerns, by probing into the relevant concepts, as well as discussing the statistical validity of our results.

W2: claiming first to show long-tailed VLM datasets, contributions with respect to prior work, and incorrect citation

We thank the reviewer for pointing out these concerns.

• We apologise for the confusion—we would like to clarify that we do not explicitly claim that we are the first to establish that VLM pretraining datasets are long-tailed. We agree with the reviewer that prior work has showcased that VLM pretraining datasets like LAION-400M are long-tailed with respect to the concepts they contain, and we build, complement and generalize the findings of these prior works.

• Contributions with respect to [1]: We emphasise that we complement this prior work and point out that our work comprehensively tests the strength of the log-linear scaling trend across several datasets, spanning varying levels of data curation and dataset sizes. Further, we note that in [1], the estimated frequencies are computed using only the text captions of LAION-2B. These estimated frequencies are then used as the canonical frequencies for plotting the performance-frequency curves for all the tested models (despite these models being trained on different pretraining datasets other than LAION-2B). Our work strongly showcases why this apparent asymmetry in their frequency estimation methodology should work—from tab 4, we show that different VLM pretraining datasets are strongly correlated in their concept distributions. Hence, in spite of [1] using only LAION-2B as their source dataset for freq estimation, their results roughly hold true because of this strong correlation across pretraining datasets. Our methodology of incorporating both images and text captions when computing the freq estimates is crucial for explaining this. Hence, we believe that our work comprehensively generalizes and explains the findings of prior work while also providing insights into the pretraining datasets (eg, misalignment degree and correlation of concept distributions in datasets). We will be sure to clarify this in the revision.

• Scale of experiments: While we agree that the largest dataset we analyse (LAION-400M) is smaller than LAION-2B/DataComp-1B, we point to fig 3 and 22 of [2], where authors showcase that zero-shot performance of different data curation methods are strongly correlated across different dataset scales (12.8M vs 128M vs 1.28M), while the source distribution is held constant (Common-Crawl). Similarly, in our case since LAION-400M, LAION-2B and DataComp-1B are sourced from the same Common-Crawl source-distribution, we are confident our results also generalize to LAION-2B / DataComp-1B.

• Incorrect citation: We apologise for this oversight—we fixed it in the paper to reflect the correct version.

Thank you for your response—it has addressed most of my questions. Here are my thoughts on each point:

1. I think it's important to note the ambiguity problem as a limitation of the current frequency measurement pipeline. This can help offer some explanation of the dips observed and clear some questions a reader may have when observing the dips.

2. It seems there's still some confusion regarding [1], which also analyzes LAION-400M, not just LAION-2B, as I mentioned in my review. Since your work is closely related to previous research, it's crucial to provide a clear comparison that differentiates your contributions from existing work. I suggest enhancing the current literature review section to better highlight these novel aspects, as the current writing doesn't achieve this.

3. Thank you for detailing the dataset curation process. However, I'm curious if you considered reducing the benchmark size by sampling only classes with fewer than 1,000 instances. If so, did this result in further accuracy degradation?

4. It would be beneficial to include the dataset statistics and high-level insights presented in the rebuttal within the appendix of the submission.

I have asked one additional question above and will wait for the authors' reply.

Q1: Dips in accuracy at high freqs

Thank you for the suggestion. We look into CC-3M and CC-12M, the pretraining datasets corresponding to which we see dips in accuracy on the classification tasks. From our analysis, we hypothesise two main reasons for these performance dips:

• Concept ambiguity: we observe many concepts that are homonyms / polysemous (same spelling but different meaning ie can represent multiple concepts at once). Some examples are watch, bear, house, fly, bridge, cloud, park, face, bar, tower, wave, etc.
• Broad concepts: A concept with a broader scope of definition supersedes a narrower one (concept 'dog' vs the specific breeds of dogs seen in imagenet-r ('yorkshire terrier', 'boston terrier', 'scottish terrier', 'golden retriever', etc)). These concepts are too coarse-grained and hence can be visually represented by a diverse set of images. Performance variance of these concepts can be quite high based on the specific set of images given for testing.

These ambiguities become more prevalent the more ubiquitous a concept is, which is directly tied to its frequency obtained from pretraining datasets. Some more examples for a deeper understanding of the diversity of concepts are: cucumber', 'mushroom', 'Granny Smith', 'camera', 'chair', 'cup', 'laptop','hammer', 'jeep', 'lab coat', 'lipstick', 'american-flag', 'bear', 'cake', 'diamond-ring', etc. We will provide this analysis in the appendix and hope this adequately addresses the question.

Q2: Variance in fig 2 points

Thank you for the insightful question. We provide zero-shot classification plots for CC-3M, CC-12M, and LAION-400M in the uploaded rebuttal PDF, including 95% confidence intervals for each point. This approach follows the standard practice from works like [1, 2]. Our plots show that the spread at higher frequencies is significantly larger than at moderate frequencies, following the finding in the previous point that higher frequency concepts are more ambiguous and polysemous. These results support the observed dips in accuracy at high-frequency points. We will include these plots in the appendix and discuss this in the paper.

Q3: Zero-shot perf predictive of few-shot/linear-probing?

Thank you for this insightful question. Our work focused on zero-shot evaluations, the standard for assessing vision-language models "out-of-the-box". However, according to Gadre et al. fig. 16, ImageNet zero-shot performance and linear probing performance are highly correlated across various CLIP models. Thus, it is likely our trends would also apply to few-shot fine-tuning, at least for ImageNet.

We agree with the reviewer that investigating how log-linear scaling trends change with few-shot fine-tuning, such as in TIP-Adapter, CLIP-Adapter, or CoOP, would be a valuable follow-up to our work. We will add a point on this in our future works section in the appendix.

Q4: Training time for models?

Each of the models we consider are roughly trained for ~30 epochs each. This results in a different total compute budget (samples seen budget) for each run. We provide some canonical estimated total GPU hours for training different models below:

Thank you for your further investigation and thorough response! My questions are nicely addressed, and I really enjoyed reading your paper. I am going to keep my score as it is, since I already gave a high score from the beginning.

W1: Why not simply scale up models?

Thank you for raising this point.

• Train-test similarity as a control factor. We agree models with higher ImageNet accuracy perform better on the long-tail. Note however that results in fig 6 are not normalized for “train-test similarity”. Normalizing for train-test similarity is crucial for interpreting these results. For eg, the absolute performance on low-freq concepts is higher across larger datasets like LAION400M in fig 2. However, when normalized for train-test similarity (fig 4 left), performance drops to levels comparable to smaller datasets like CC-12M. This highlights the importance of considering train-test similarities when comparing absolute performance across pretraining datasets.
• On scaling up models. We agree increasing model sizes can improve performance offsets, as supported by scaling laws. However, trends in performance degradation on low-freq concepts remain consistent across different model-sizes (RN50 - ViT-L/14), indicating a significant issue even with larger models.

We will clarify these points in the manuscript and add a discussion on model scaling laws.

W2: What are panels in Fig 3/Fig 5?

We apologize for the lack of clarity.

• Figure 3: The different panels denote log-linear performance-frequency scaling curves for different T2I models. Note that in total we analyze 24 different T2I models (see tab 2). Since it would be tedious to fit all 24 onto a single plot, we split into 8 sub-plots with 3 models each.
• Figure 5: The different panels in fig 5 all showcase the concept count distribution across three pretraining datasets. In each plot, we showcase estimated frequency of concepts in the pretraining dataset on the y-axis, and the sorted concept index (sorted by frequency) on the x-axis. Since there are three ways to estimate concept frequencies (freqs from text captions, freqs from images, and joint frequencies from both images and text captions), we showcase the frequency distributions as obtained using all three methods independently.

We will update the paper to add these clarifications, and hope this will simplify and ease the presentation of results.

W3: Paper does not really study 0-shot performance, no points with freq 0 in figs 2 and 3

Thank you for this important point. The reviewer is right—we explicitly exclude zero-freq concepts from our evaluations following [1], since frequency estimation is potentially noisy, leading to low recall rates (also discussed in appx H). However, to verify if our log-linear trends still hold when including all the zero-freq concepts, we replot all our main zero-shot classification results from fig 2 by including the ones which have zero-freqs—this plot is in the attached PDF. We find our main log-linear scaling trends from fig 2 are retained. To further corroborate this, we present average accuracies for concepts with freq 0 and non-zero freq bins in the table below. We note that average performance for the 0-freq concepts are significantly lower than other non-zero freq concepts, especially when compared to very high-freq concepts. This justifies our main claim that exponentially more data is needed per concept to improve performance linearly.

W4: Issues with RAM++

Thank you for this important point.

• Extensive Ablations on Image-Tagging Models: We agree using a pretrained model for tagging concepts might introduce biases. However, we conducted extensive ablations on this (see appx H). We tested concept-tagging ability of open-world object detectors (Owlv2) and multi-label tagging models (RAM/RAM++), finding RAM++ to be most precise for our case (see appx H.1, fig 20).
• Context Enhancement improves RAM++ tagging precision: Unlike object detectors, RAM++ leverages GPT-4-generated descriptions (see tab 5 appx), improving tagging precision by using visual descriptions to better identify concepts (this has been shown to enhance performance [2,3].
• Robustness vs RAM++ Thresholds: We investigated different hparam thresholds for RAM++ (appx H.2). Despite some thresholds yielding sub-optimal tagging, our log-linear scaling results remained robust.
• Human verification for misalignment results: To verify misalignment results, we manually annotated 200 random image-text pairs from each dataset as aligned or misaligned. An image-text pair is misaligned if the text caption was irrelevant to the image. Previous work also found a similarly small random subset over large-scale web-datasets to be representative [4]. Our estimated misalignment results from tab 3 were in line with human-verified results (see tab below), corroborating our findings. |Dataset|tab 3 results|human baseline results| |-|-|-| |CC-3M|16.81%|18%| |CC-12M|17.25%|14.5%| |YFCC-15M|36.48%|40.5%| |LAION-400M|5.31%|7%|
• High YFCC Misalignment Degree: From our human experiment, we found that the high misalignment degree in YFCC-15M is likely due to the lack of text quality filtering. YFCC-15M images are sourced directly from Flickr, where captions often provide high-level context rather than accurately describing the image content.

We hope these clarifications address the reviewer's concerns and provide a better understanding of our work.