Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets

We thank the reviewer for time dealing thoroughly with our work. We appreciate the reviewer’s remarks on the clarity and writing quality of the paper, highlighting the effort spent for methodologically rigorous and reproducible experiments, also mentioning usefulness for the open-source community. In following, we would like to address the relevant points raised by the reviewer:

1. The scope of this paper is mostly focused on contrastive and captioning objectives - it would be interesting to see results on other alternative architectures and objectives.

Available compute budget for the study allowed us to demonstrate execution of validated, reproducible comparison using scaling laws derivation for important architectures such as MaMMUT, CLIP, SigLIP, CoCa, CaP and for open datasets DataComp, Re-LAION and DFN (for DFN, we executed additional experiments for the rebuttal that will enter the revised manuscript). Comparison of further learning procedures requires additional compute budgets, and we hope that the introduced method will be used in follow up works to perform further useful, reproducible comparisons, guiding validated search for stronger scalable learning procedures.

2. Evaluation primarily limited to zero-shot settings, with minimal analysis of few-shot learning, fine-tuning transfer, or vision-language understanding tasks

Choosing zero shot evaluation for scaling laws (with exception of dense segmentation task where we performed fine-tuning) was due to two considerations. First is grounded in works like [1] and [2] stating that zero shot performance is a strong predictor for fine-tuning performance, and thus can be taken as good proxy for model quality. That is, stronger scalability for zero-shot would highly likely imply stronger scalability for full fine-tuning. Second comes from available compute budget considerations: zero shot evals are cheap in compute, while full fine-tuning is expensive, especially given the large number of obtained points for scaling law derivation it has to be executed for. We thus think that zero-shot evals provide a strong proxy for model quality on the one hand, while allowing to keep compute costs down for evals on the other hand. In follow up studies, given further compute budgets, it will be interesting to perform fine-tuning evals for control as well.

[1] Robust Fine-Tuning of Zero-Shot Models, Wortsman et al, CVPR 2022, arXiv:2109.01903
[2] Reproducible Scaling Laws for Contrastive Language-Image Learning, Cherti et al, CVPR 2023, arXiv:2212.07143

3. There is also some compute accounting ambiguity, especially when comparing architectures with inherently different parameter counts (Table 12), which raises questions about compute comparison methodology.

We would like to clarify that there is no ambiguity wrt compute accounting. openCLIP profiler routine (relying on fvcore library) computes exact FLOPS for each model scale employed in our work. The calculated compute is then used for scaling law derivation and for reporting GFLOPS as listed in Table 12, taking into account image resolution, patch size, text context length and number of parameters in vision and text towers (compute does not depend on param count only!). This avoids any ambiguity, assigning for each model and samples scale its respective compute. The calculations can be reproduced and checked by installing openCLIP and calling profiler, eg python src/open_clip_train/profiler.py --model mammut_ViT-B-32 --results-file compute_profiling.csv for any given model config.

4. … limit training to less than 3B samples to avoid repetition effects … constrains conclusions about truly large-scale behavior. When trained on 12.8B samples, performance falls below predictions. How can we be certain that this under-performance is actually due to high repetitions?

Deriving scaling laws based on measurements up to 3.07B samples seen allows us to make predictions for behavior of compute-optimal training on unique or low repetition (2.2x for 1.4B) samples. We provide evidence that these predictions are accurate using held-out points on compute-optimal Pareto front for both CLIP and MaMMUT in App. Tab. 7, as we see actual measurements on compute-optimal Pareto front falling well into the 95% confidence interval for the predictions on held-out points. Predictions for MaMMUT extrapolate 2.4 and 5.5 compute factor beyond the fit (taking points up to $C^{MaMMUT}_{\text{threshold}} = 2.59\cdot 10^{11}$ GFLOPS for the fit). Factor 5.5 would allow us to extrapolate from 3B samples seen to 16.5B samples, which is well above 12.8B. As we confirm scaling law derivation giving accurate predictions when extrapolating far ahead, the most likely explanation for the deviation from predictions on 12.8B scale is thus sample repetition, as it is also known to cause diminishing returns from previous works [3], [4]. Further reason for observed performance drop can be also due to training procedure not residing on compute-optimal Pareto frontier. For rebuttal, we train in addition openMammut B/32 on 12.8B samples of DataComp-1.4B, resulting in following overview on 12.8B scale:

For both B/32 and L/14, the measured IN1k 0-shot accuracy is below the predictions for unique samples, due to 9x sample repetitions (1.4B unique samples, 12.8B total), as elaborated above. The measured performance can be still projected from the predictions for unique samples by taking offset of 2% for MaMMUT and 1% for CLIP downwards from the confidence interval of the predicted performance on unique samples. Testing on unique samples is currently not possible, as it requires open datasets with scale of at least 6B unique samples to keep repetitions low. For accurate prediction on repeated samples, follow up work is required to derive scaling laws with corrections for sample repetitions which requires additional training experiments similar to [3], [4]. Important for our work is that comparison is valid also at larger scales, MaMMUT consistently outperforming CLIP as predicted from scaling laws, also for high sample repetition.

[3] Scaling Data-Constrained Language Models. Muennighoff et al, NeurIPS 2023, arXiv:2305.16264
[4] Scaling Laws for Data Filtering. Goyal et al, CVPR, 2024 arXiv:2404.07177

5. … SigLIP shows no benefits over CLIP … How were hyperparameters chosen to ensure fair comparison across architectures and scales? … How can we be confident that the observed scaling differences reflect architectural properties rather than differences in effort invested in optimization?

For our main comparison showcase, CLIP vs MaMMUT, wide hyperparams interval was scanned for both models extensively, resulting in 672 scans for CLIP and 1010 scans for MaMMUT on DataComp-1.4B. We invested more effort into MaMMUT tuning as CLIP architecture was already extensively tuned in previous works and we could afford to tune CLIP based on already known hyperparam settings, which is not the case of MaMMUT. MaMMUT as the previously less optimized architecture was thus tuned more to avoid unfair advantage for CLIP. Confirming there is no tuning advantage, predictions from scaling laws on held out points have same high accuracy and low uncertainty for both CLIP and MaMMUT, also showing similar RMSE (App. Tab. 7, as also elaborated above).

Further, we conduct additional experiment to double check whether there is any difference in obtained scaling law if working with same number of points for MaMMUT as for CLIP on DataComp-1.4B. We perform bootstrapping, sampling randomly 672 points from 1010 available points for MaMMUT, doing 10 trials, fitting scaling law for each trial and averaging the obtained scaling law coefficients. We observe that the obtained fit coefficients have no significant difference from scaling law obtained with 1000 points:

Thus, comparison on reduced points shows again same trends. Measurements are balanced for Re-LAION (750 vs 703 points), as well as for DFN (737 vs 732 points). Comparison via scaling laws on Re-LAION (Fig. 2) and DFN gives same trends as on DataComp, showing MaMMUT stronger scalable than CLIP, again confirming no influence of tuning on comparison.

For SigLIP and CLIP, we also have balanced tuning situation on DataComp-1.4B with 590 tuned points for SigLIP, and thus can rely on the scaling laws derived for the comparison (Fig. 8a). Executing controlled comparison on same open dataset DataComp 1.4B is sufficient to see that replacing softmax with sigmoid transfer function does not provide any benefit over CLIP. Original SigLIP study featured a closed dataset (WebLI), such that controlled comparison there is not possible. Our result shows that original SigLIP strong performance is highly likely due to the closed dataset WebLI, and not due to replacing softmax with sigmoid transfer function as claimed in the original work.

We are delighted to hear our response was helpful to clarify the questions and thank the reviewer for engaging with timely reply and for raising the score.

As reviewer is interested in the consistency of the comparison favoring MaMMUT vs CLIP across scales, we provide additional summary across selection of reference scales, showing actual measurements from the performed experiments for the 3 datasets (Re-LAION, DataComp, and DFN - addition for the rebuttal) on the Pareto front of best tuned models, with exact numbers for MaMMUT advantage (the "-" entries correspond to missing measurements, where also comparison is not possible)

We will update the manuscript accordingly with additional material and thank again for the discussion that was helpful to further clarify and improve the work.

Thank you to the authors for the thorough response. My main concerns have been addressed.

The extensive hyperparameter search and scope of experiments demonstrate appropriate rigor given the substantial compute investment. I am satisfied by the additional results in point 4 showing that the scaling laws remain valid and MaMMUT continues to outperform CLIP at larger scales. Additionally, the clarification regarding compute accounting using the openCLIP profiler methodology resolves my misunderstanding about fair comparison across architectures.

I also think the SigLIP analysis suggesting that performance benefits may be dependent on WebLI, rather than architectural, is interesting and merits further investigation.

Given the solid empirical contributions and valuable open-source resources this work will provide to the community, I will raise my score.

We thank the reviewer for time dealing thoroughly with our work. In following, we would like to address the relevant points raised by the reviewer:

1. Although the code release has been mentioned in the paper, what about the pre-trained models? The reviewer assumes these pre-trained models will also be made public.

This is correct. We will release the pretrained models (for the rebuttal, we additionally trained DataComp 12.8B openMammut B/32 which will complement openMammut L/14 already described in the study) from all points measured for scaling law derivation (in total over 4500 models trained on DataComp, Re-LAION and DFN [1], for which we conducted further scaling law derivation experiments for the rebuttal), including intermediate training checkpoints for each model, their evaluation and source code to reproduce training, evaluation and fitting of scaling laws. In the table below is the overview of all points measured during the study (including experiments on DFN [1] performed for the rebuttal) for which models will be released.

[1] Data Filtering Networks. Fang et al, ICLR 2024, arXiv:2309.17425

2. … since this paper uses OpenMaMMUT which involves not only CLIP-style contrastive learning but also sentence generation, it would be beneficial to show examples of the kinds of captions generated for various images …

We agree that it will be a good reminder that openMaMMUT is also a functional captioner by providing examples of caption generation. Main focus of the study was to demonstrate the capability of our method to compare pure contrastive CLIP with contrastive+captioning loss mixture of MaMMUT, and check where scaling law based comparison can identify which procedure is the more scalable in this fully controlled setting. Thus, adding captioning loss served the purpose of demonstrating comparison between two controlled learning procedures, and while we gather evidence that contrastive+captioning loss mixture is what makes openMaMMUT perform better at larger scales on various downstream tasks compared to CLIP, we do not assume strong function of captioning itself. However, it is indeed helpful to present captioning examples nevertheless. In our additional experiments, we see examples of good captioning, eg correctly identifying multiple objects on the image or performing simple OCR from written text, as well as examples of failure, for instance not being able to parse calligraphic writing. We will update the revised manuscript with corresponding material and also provide a openMammut based demo for captioning on HuggingFace Spaces to try captioning on any dropped image.

3. Similar contributions such as Molmo & PixMo (CVPR 2025) aim to open-source VLMs … discussing their positioning and relationship would enhance paper's overall value …

We have a different focus than works like Molmo & PixMo, in 1) describing a novel generic method for robust comparison of various learning procedures (models & datasets) across scale span, while the mentioned work presents a single model and dataset showing performance on few selected scale points 2) dealing with fully controlled scenario of single stage pre-training from scratch without relying on already pre-trained components, which makes training procedure and comparison fully reproducible, in contrast to Molmo which eg uses pre-trained language model and pre-trained vision encoder (with data for vision encoder being closed, making whole procedure not fully controllable and reproducible). In follow up works, an interesting direction would be to apply scaling law based comparison to VLM post-training setting with fully open pre-trained components like studied in our work. This would enable VLM training procedure comparison that takes properly into account all stages, without having closed components (models & datasets) introducing non-controllable confounds. We will extend discussion in revised manuscript with the outlook to future work on scaling laws based comparison for multi stage VLM learning procedures with pre-trained components.

4. … pre-training approaches … specialized for zero-shot classification, retrieval, and segmentation? If the underlying vision-language tasks share common characteristics, focusing on contrastive learning alone could suffice …

In the study, we look at CLIP as a model using contrastive loss only and at MaMMUT as a model using contrastive+captioning loss to provide a compelling showcase for comparing two learning procedures using our method based on scaling law derivation across different settings. The comparison allows us to predict which of the procedures provides better performance at larger scales. It turns out to be MaMMUT that shows stronger scalability and outperforms CLIP at larger scales. Using our comparison method, we can thus decide whether contrastive learning alone could suffice to perform strongly on those downstream tasks, which turns out to be not the case – as according to scaling law derivation based comparison, contrastive+captioning loss mixture performs consistently better across all the downstream tasks from certain compute threshold on (while on smaller scales on the other hand, contrastive learning via CLIP prevails), and thus poses the better, stronger scalable alternative to contrastive learning alone.

hi reviewer, thank you for spending time to read the paper and provide feedback. The authors have provided responses to your feedback and please spend some time to read the responses. Please do make sure let the authors know whether you still have any concerns, questions, or suggestions after reading their responses.

We thank the reviewer for timely engagement and are glad to see our comments were helpful. The anonymous repository in the link https://anonymous.4open.science/r/open_clip_scaling-76D2/README.md (that was already part of the original submission) contains the code to reproduce the plots from the raw data, the collected raw data itself and also the patches that can be applied to publicly available open source openCLIP and CLIP Benchmark versions which allows executing all the experiments reported in the paper including training openMaMMUT. The full version will contain all the full source code necessary for reproducing whole experimental pipelines involved in scaling law measurements and derivation, including training experiments and model evaluation.

We thank the reviewer for time dealing thoroughly with our work. We appreciate the reviewer’s remarks on the excellent quality of the paper, emphasis on rigorous and reproducible experiment execution and on contributions to the open-source community. In following, we would like to address the relevant points raised by the reviewer:

1. Thanks for careful reading and catching the typos. At line 216, we will remove the incorrect minus sign for $\alpha$, reading "($\alpha=0.208$ vs. $\alpha=0.354$)". At line 186, Re-LAION-1.4B will be replaced by DataComp-1.4B, reading "... stronger scaling for DataComp-1.4B on classification ..."

2. the question investigated in the paper is specific to the two models and the two datasets …

The general question our work addresses is how to perform robust, reproducible comparison of training procedures for open foundation models, both with regard to architectures and datasets involved. To showcase how such comparison based on scaling law derivation can be conducted, we choose CLIP and MaMMUT and open datasets DataComp-1.4B and Re-LAION-1.4B as an example study subjects – the posed question and method are though not specific to the models and datasets we took in our example. Already in this study, we look at further models, deriving scaling laws for comparison of CLIP and MaMMUT with SigLIP (App. Fig. 8 (a)), CoCa (App. Fig. 8 (b)), CaP (App. Fig. 17). For the rebuttal, using the same method, we conduct additional experiments on another important open reference dataset, DFN-1.4B [1], obtaining further model and dataset comparison on DFN-1.4B. We again obtain same trends from scaling laws derived on DFN-1.4B as observed for DataComp (Fig. 1) and Re-LAION (Fig. 2) – MaMMUT being stronger scalable than CLIP, providing further support for consistency of the comparison across datasets and downstream tasks. Dataset comparison reveals DFN-1.4B [1] leading to better training for both CLIP and MaMMUT than DataComp and Re-LAION. This shows that our method is generic and can be applied for robust model and dataset comparison in various scenarios. We will update the revised manuscript with corresponding material.

[1] Data Filtering Networks. Fang et al, ICLR 2024, arXiv:2309.17425

3. In Figure 8, the paper shows the comparison of SigLIP, CoCa, CLIP, and MaMMUT on the DataComp-1.4B dataset. Everywhere else, the paper also compared the Re-LAION-1.4B dataset. Does the same pattern hold on Re-LAION-1.4B dataset?

Large fraction of available compute budget for the study was used to obtain measurements for the main comparison for CLIP and Mammut, deriving scaling laws and showing consistent trends across datasets and downstream tasks (Fig. 1, 2). Across datasets, we executed 4604 measurements (full training runs) to derive scaling laws for both models, see summary table below:

As compute cost for executing dense measurements for scaling law derivation is substantial (see App. Tab. 4), the budget was not sufficient to repeat the same extensive measurement procedure across datasets for further architectures, comparison for which was thus restricted to DataComp-1.4B. To show consistent patterns for other datasets, we conduct for the rebuttal additionally for the main comparison showcase in our study - CLIP vs MaMMUT - comparison of data efficiency on Re-LAION-1.4B, which confirms observations on DataComp. and complement this with another data efficiency comparison for DFN-1.4B. Both confirm the patterns observed on DataComp - MaMMUT being a more data efficient model than CLIP. Together with the additional experiments on DFN-1.4B described above, we can confirm the same pattern of consistent comparison outcomes for CLIP vs MaMMUT, pointing to MaMMUT as stronger scalable and more data efficient learning procedure than CLIP across all studied open datasets. We will update the revised manuscript with corresponding material.

4. why MaMMUT has better scalability than CLIP … fundamental question of how to design a more scalable model … study on whether it is the training loss or the model architecture that makes MaMMUT more scalable.

Evidence obtained from comparisons in our work allows us to draw conclusions about what makes MaMMUT scale stronger than other measured architectures. On the one hand, we have a comparison to CLIP, which has in contrast to MaMMUT only contrastive loss, thus ablating captioning loss and cross-attention from vision to text tower. On the other hand, we also compare to Cap (see App. Fig. 17 and App. Sec. E “Scaling behavior of other architectures”; for rebuttal, we additionally perform comparison MaMMUT vs Cap using log likelihood evaluation based on logprobs of the text tower), which has captioning loss and cross-attention, but no contrastive loss, thus ablating for contrastive loss. As both CLIP and Cap underperform MaMMUT especially at larger scales, we can conclude that neither contrastive loss nor captioning loss + cross-attention alone are responsible for providing stronger scalability, and it is the contrastive and captioning loss mixture which creates more scalable training procedure (see also App. Sec. E.).

In our study, the comparisons between the various models are performed in a common reference frame (eg same open datasets) and across scales, while fully controlling for adding and removing components of the learning procedure for each scaling law derivation. This allows us to identify the effects of various ablations on learning procedure scalability. This is in contrast to other works, where on the one hand, unknown confounds like closed datasets make it hard or impossible to infer which of the differences between the learning procedures (algorithmic or dataset based) led to better performance, while on the other hand performance is measured only on few selected scales without estimating scaling trends, making unclear whether comparison holds for wider span at larger scales. Designing better learning procedures requires an approach to conduct guided search for more scalable procedures given the existing ones as reference baselines. This approach requires a robust method for training procedure comparison valid across scales that can operate in the same reference frame to control for any interventions - including controlling the datasets which requires open data, - and this is where our study makes progress. We will update the revised manuscript with corresponding material and make our conclusions about why MaMMUT is a more scalable learning procedure clearer in main text’s discussion section.

We thank the reviewer for time dealing thoroughly with our work. We appreciate the reviewer’s high appraisal of the study, some of the summaries are so precise and well written that we consider adapting it for our own communication. In following, we would like to address the relevant points raised by the reviewer:

1. … The core comparison is between only two architectural paradigms: CLIP and MaMMUT … the paper's title and claims suggest a more general framework for comparing "Open Foundation Language-Vision Models."

The general question our work addresses is how to perform robust, reproducible comparison of training procedures for open foundation models across scales, both with regard to architectures and datasets involved. We choose CLIP and MaMMUT and open datasets DataComp-1.4B and Re-LAION-1.4B as an example study showcasing how our method works for comparing important vision-language learning procedures. The method is though not specific to the chosen study subjects (and is also not restricted to language-vision scenario only) and can be applied to compare any learning procedures that feature open foundation models and open datasets by conducting scaling law derivation as described in our work.

2. The study does not include other important architectural variants, such as (1) SigLIP, which uses a different loss function (sigmoid loss) and has shown very strong performance …

We would like to clarify that we do look at SigLIP in our study, deriving scaling law on DataComp-1.4B for this architecture, where softmax in CLIP is replaced by sigmoid transfer function, and comparing with original CLIP architecture (App. Sec. E, “Scaling behavior of other architectures”). By executing controlled comparison on same open dataset DataComp 1.4B we obtain evidence that replacing softmax with sigmoid transfer function does not provide any benefit over CLIP, as we do not see any difference in scaling trends obtained from scaling laws derived on same open dataset (App. Fig. 8 (a), App. Sec. E). Original SigLIP study featured a closed dataset (WebLI), such that independent controlled comparison with CLIP there is in contrast not possible. Our result shows that strong performance reported for SigLIP previously is highly likely due to the closed dataset WebLI, and not due to replacing softmax with sigmoid transfer function as claimed in the original work.

3. … (2) BLIP and BLIP2, which are generative large Language-Vision Models, (3) LLAVA, QwenVL, etc which incorporate CLIP encoder with large LLM …

While methods from our study can be applied to compare various foundation models and datasets, we have deliberately chosen a scenario where 1) pretraining is happening from scratch from randomly initialized models, excluding confounds bound to taking already pre-trained components 2) pretraining procedure is simple single stage . This allows for fully controlled setting to perform scaling law derivation, providing same reference frame for the comparison of learning procedures. Learning procedures that lead to BLIP3, Qwen-VL and other similar works are using various pre-trained components, often trained under unknown conditions and performing training in various multiple stages, which necessitates further hyperparam and dataset decisions per stage. This is in contrast to our showcase, where all components are fully controlled, open and hyperparams can be tuned in the single stage, allowing to derive scaling law for the full extent of a given learning procedure to serve as base for validated, reproducible comparison to other procedures. Another consideration is available compute budget – each comparison requires scaling law derivation and we have chosen to use the available compute budget to demonstrate our method in clear way for comparison of important reference vision-language learning procedures CLIP and MaMMUT. We hope that our method can be used to compare other important learning procedures in search of more scalable open foundation models in follow up works, taking additional necessary compute required to execute such comparisons.

4. Is it possible that the chosen hyperparameter search space or the final selected parameters favor the MaMMUT setup? … a skeptic could argue that the CLIP baseline might not have been trained under its own optimal conditions, thus artificially enhancing MaMMUT's apparent scaling advantage.

For our main comparison showcase, CLIP vs MaMMUT, we used wide intervals to scan for both models hyperparams extensively. We provide here overview for the hyperparam intervals and resulting scans for both models on datasets used in the study (for rebuttal, we also performed experiments on another important open dataset, DFN-1.4B):

As evident from the table, the measurements are balanced for Re-LAION (750 vs 703 points), as well as for DFN (737 vs 732 points). For DataComp, we have 672 scans for CLIP and 1010 scans for MaMMUT on DataComp-1.4B. More measurements were performed on DataComp for MaMMUT to avoid unfair advantage for CLIP, as CLIP architecture was already extensively tuned in numerous previous works, which is not the case for MaMMUT that was not open sourced before and was not available for tuning by the broad community as was the case for eg openCLIP. Confirming there is no tuning advantage, predictions from scaling laws on held out points from DataComp have same high accuracy and low uncertainty for both CLIP and MaMMUT, also showing similar RMSE - App. Tab. 7, here a short version:

The predictions are accurate using held-out points on compute-optimal Pareto front for both CLIP and MaMMUT, as we see actual measurements on compute-optimal Pareto front falling well into the 95% confidence interval for the predictions on held-out points. (For the fit, points up to $C^{MaMMUT}_{threshold} = 2.59\cdot 10^{11}$ GFLOPS for MaMMUT and up to

$C^{CLIP}_{threshold} = 4.07\cdot 10^{11}$ GFLOPS for CLIP were taken, which extrapolates for MaMMUT 5.5 and for CLIP 2.8 times; see App. Tab. 7). Similar high prediction accuracy for both models again hints on absence of tuning advantage for either CLIP or MaMMUT.

Further, we conduct an additional experiment to double check whether there is any difference in obtained scaling law if working with the same number of points for MaMMUT as for CLIP on DataComp-1.4B. We perform bootstrapping, sampling randomly 672 points from 1010 available points for MaMMUT, doing 10 trials, fitting scaling law for each trial and averaging the obtained scaling law coefficients. We observe that the obtained fit coefficients have no significant difference from scaling law obtained with 1000 points:

Thus, comparison on reduced points shows again the same trends. Further, comparison via scaling laws on Re-LAION (Fig. 2) and DFN (will be included in the revised manuscript) which have balanced number of scanned points gives same trends as on DataComp, showing MaMMUT stronger scalable than CLIP, again confirming no influence of tuning on comparison. We can thus conclude from the gathered evidence that observed trends and performed comparison are valid and are due to differences in learning procedure (MaMMUT having contrastive+captioning loss mixture vs contrastive only for CLIP), and cannot be due to differences in tuning. We will update the revised manuscript with corresponding material.

We thank the reviewer for timely engagement and are glad to hear that our response was helpful to solve concerns.

As reviewer expressed specifically interest in impact of tuning on observed performance across scales and datasets, we provide additional summary across selection of reference scales, showing actual measurements from the performed experiments for the 3 datasets on the Pareto front of best tuned models, together with their tuned hyperparams. We see that Mammut consistently outperforms CLIP, while tuned hyperparams for both often have same values. This provides further evidence that tuning for both converges to very similar hyperparam regions and does not give advantage or disadvantage for any of the studied architectures.

IN1K downstream performance (the "-" entries correspond to missing measurements, where also comparison is not possible)