Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work is “well-written”, contains “extensive experiments” and “good performance”, and will “definitely benefit the community” when open-sourced. We summarize your questions and provide responses below:

W1&3: Findings are well-known and verified from previous work

A: We believe the study of the Multimodal Large Language Model contains many moving parts, and previous works provide isolated studies on each component. This potentially leads to contradictory results in different studies (e.g., freeze or unfreeze vision encoder in Prismatic VLM vs. Deepseek VL). Our work undertakes a systematic study, isolating and studying the numerous moving parts, which we believe allows more long-standing conclusions that benefit the community.

Further, we respectfully argue that our work differs from previous works in the following ways:

• More Comprehensive Experiments: Our study considers more vision backbones and organized benchmarks than previous works, yielding new insights. For example, we find that high-resolution ConvNets benefit OCR & Chart tasks, and self-supervised learning models can potentially compete with language-supervised ones. Our experiments also reveal the properties of different CLIP models (e.g., SigLIP, EVA-CLIP) beyond ImageNet accuracy, providing valuable insights for both MLLM and Vision model developers. For instance, we observe that EVA-CLIP performs well in most domains but struggles with Chart tasks. This highlights the need for CLIP developers to focus more on OCR & Chart data collection during training.
• Findings on high-res encoders: While concurrent works emphasize the importance of high-resolution features, we take a step further to pinpoint the potential of using ConvNets, such as the ConvNext OpenCLIP model, to efficiently and effectively process high-resolution images.
• New Vision Model Aggregation Strategy: Compared to previous works that focus on fusing vision models, we study both which models to combine (§3.5) and strategies for combining them (§4). Our SVA approach preserves the resolution, maintains the spatial inductive bias in images, and uses a fixed number of visual tokens.

We also thank the reviewer for raising this point and suggesting these works. We will discuss each of them in our revision.

W2: Training only using proposed tuning datasets, compare with LLaVA 1.6

A: We first want to clarify that LLaVA 1.6 (LLaVA-Next) proposes a dynamic resolution approach using 2880 tokens, while our SVA module uses only 576 fixed tokens. We conduct additional experiments with the LLaVA model trained using LLaMA-3-8b and Cambrian-7M data. Due to the short rebuttal period, we use the conventional 576 visual tokens in LLaVA and LLaVA-1.5, not the dynamic high-resolution proposal of LLaVA-Next. Despite fewer tokens, this version matches or outperforms LLaVA-Next in General, Knowledge, and Vision-Centric tasks. Adding our SVA module further improves performance, especially in OCR & Chart tasks, while still using only 576 tokens.

W4: Several figures are useless and common knowledge for MLLM (e.g, Figs. 1 & 2)

A: We hope our work provides a systematic study around MLLMs and can serve as informational for audiences both within and beyond the MLLM community. Especially now, as MLLM is becoming an ever-growing community, the introduction and figures serve as preparation and context-setting for a broader audience. We make no claims of novel findings in the initial figures and reserve such insights for the later sections after providing the audience requisite context. Nevertheless, we thank the reviewer for raising this concern, and we will consider condensing our presentation in the revised version.

W5: Study of SVA Module is not enough

A: We thank the reviewer for raising this crucial point. We have added a study of the importance of visual features from different vision models to different image categories by investigating the attention score distribution in our SVA module.

We evaluate our Cambrian-8b model on GQA, DocVQA, and ScienceQA (representing three different benchmark categories), and tabulate attention distributions below. We can see that on real-world images (GQA), the contribution of different vision models is relatively uniform, in part due to the similar characteristics of SigLIP and CLIP. On document-type images (DocVQA) which are text-heavy and often high-resolution, the influence of SigLIP increases and that of ConvNext greatly increases to aid in high-resolution information processing. For scientific images (ScienceQA) composed of illustrations and diagrams about different science categories, the contribution of SigLIP is further increased while the portion of DINOv2 decreases compared to GQA.

We also study the performance of our Cambrian-8b model with SVA modules on different sizes of alignment and instruction tuning data. The results are shown below. We can see that increasing the size of alignment data leads to improvement in all benchmark categories. Increasing the size of instruction tuning data leads to notable overall improvement, and the instruction tuning data is especially helpful for Knowledge, OCR & Chart, and Vision-Centric tasks.

We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work “notably well-written”, “shows the limitations of MLLM benchmarks”, and “a great contribution to this field” via our fully-open approach. We summarize your questions and provide our explanations below:

W1: Comparison and Discussion between SVA and C/D-Abstractor

A: We appreciate the valuable suggestion. We provide such discussion and analyses here, and will incorporate this into our revision.

We first emphasize that our SVA module is distinct from other spatial-based connectors (e.g., C/D-Abstractor) in its ability to dynamically combine visual features from multiple vision models with varying resolutions.

However, to isolate the effect of spatial inductive bias, we consider the case of token reduction using a single vision encoder. Specifically, we use OpenAI CLIP as the vision model and compress its original 576 tokens to 36 tokens using our SVA module and other connectors. We include three baselines:

1. Direct interpolation + MLP
2. C-Abstractor
3. LDPNetV2Projector (similar to C-Abstractor but more light-weight)

We conduct experiments with the 1.2M alignment + 0.7M instruction tuning data setting with Vicuna-1.5-7b as the language model. For fair comparison, we do not include multi-layer aggregation inside the LLM for our SVA baseline.

We tabulate results below:

Compared with the simple MLP baseline, C-Abstractor performs better on General and Vision-Centric tasks but inferior on Knowledge and OCR & Chart tasks. LDPNetV2 performs similarly to the MLP baseline. Our SVA consistently demonstrates superior performance across all categories, especially in OCR & Chart and Vision-Centric tasks, demonstrating its effectiveness in information compression.

One possible explanation for SVA's data efficiency compared to C-Abstractor is that the SVA module performs local attention on all positions in the grid with the same parameters, so our SVA module receives more supervision.

W2: Overlapping findings with existing studies

A: We believe the study of the Multimodal Large Language Model contains many moving parts, and previous works provide isolated studies on each component. This potentially leads to contradictory results in different studies (e.g., freeze or unfreeze vision in Prismatic VLM vs. Deepseek VL).

Our work undertakes a systematic study, combining different moving parts together. We hope to draw more robust and reliable conclusions and clarify the contradicting conclusions that exist in the MLLM domain. In the meantime, we aim to push the study of these modules to the extreme, especially in the fully open-source setting. For example, we carefully compare 15+ vision models, hoping to provide insights to both the MLLM and visual representation learning communities. We also collect and curate, to our knowledge, the largest open-source instruction tuning datasets. These efforts turn the findings in our work into pieces that narrow the gap between open-source and proprietary models.

W3: Finding 7

A: Regarding Finding 7, we draw this conclusion based on multiple experiments rather than a single data point. We observed that, compared to ensembling only CLIP models, adding the DINOv2 model improves performance, especially on vision-centric benchmarks like RealWorldQA and CV-Bench. We appreciate the reviewer's feedback and will include more clarification on this finding in our revision.

Q1: Question about model combination and two entries for "SigLIP+DINOv2+ConvNeXt"

A: Thank you for reviewing our work so carefully and catching this typo! The second instance of “SigLIP+DINOv2+ConvNeXt” uses a ConvNeXt-L not an XXL. We will correct this in the revised version of the draft.

L1: Privacy of Internet Web Search

A: Thank you for raising this question! We absolutely value the importance of data privacy and copyright. We respect and approach this issue in the following two ways:

Collect Data from Licensed Websites: Our web agent collects data from Wikipedia, which is licensed under CC BY-SA (https://en.wikipedia.org/wiki/Wikipedia:Copyrights). We attribute the data source in §5.1 and Appx. E.2. Fully Open Source Data Collection: We also fully open-source our data collection pipeline in Appx. E. This transparency allows for thorough inspection and verification, ensuring that our methods do not violate any copyright or data privacy regulations.

Data privacy is a collective challenge faced by the community, and proprietary models often do not disclose details about their data pipeline. One of the aims of our project is to raise awareness and inspire new research on this topic by being fully transparent. We will release all details of the data collection pipeline and plan to include a section in the revision to discuss this further.

Thank you for your clear responses. As most my concerns are addressed, I will maintain the original rating of accept.

We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work “insightful and well-motivated”, contains “rigorous and carefully designed experiments”, and “can be valuable for the future development of multimodal LLMs”.

W1 & Q1: Discussion around native MLLMs

A: We thank the reviewer for raising this question! We share our thoughts on these models below and show our findings in Cambrian are very transferable to these native models. We will add the discussion below to the revised version of our draft.

Thoughts about native MLLMs

We find native Multimodal LLMs that do not use a pre-trained vision encoder, like GPT-4o or Reka, to be an intriguing and promising approach. These native models have only recently been explored and are predominantly developed by proprietary companies such as OpenAI. As a result, their actual implementation, architecture, and training methods remain largely undisclosed. Additionally, there is insufficient evidence to assert that native MLLMs can overcome the limitations of current MLLMs, such as their visual deficiencies. On the other hand, vision-only representation learning itself is a significant and meaningful objective. Our study connects this goal with multimodal learning, providing scientific insights that complement future advancements in multimodal systems, whether they are native or not.

We note that one major downside of native MLLMs is that they require much more data and computational resources, as they do not rely on the knowledge embedded in pretrained encoders.

Findings in Cambrian transfer to native MLLMs

We believe the findings and contributions in our work will continue to hold and guide the development of future models, including “native” MLLMs. Our findings regarding Data, Connectors, Evaluation, and Vision Backbones can be transferred in the following ways:

• Data: The pool of instruction tuning data and our data curation studies can be very useful for supervised fine-tuning “native” MLLMs. Training native MLLMs is likely to still consist of pretraining, supervised fine-tuning, and RLHF. The data collection and curation insights can play an important role in both the pretraining and supervised fine-tuning stages.
• Connector: The SVA module we proposed can be part of, or inspiration for, future native MLLM designs. The conflict between high-resolution features and a constrained number of tokens is likely to continue in native MLLMs. Therefore, the SVA module could be a competitive candidate for resolving this issue in such native MLLMs.
• Evaluation: Our study on the “Multimodality” of benchmarks can help native MLLMs better assess their visual capability. The categorization of benchmarks also provides more organized and interpretable evaluation protocols for future works, especially in vision-centric domains.
• Vision Backbones: Our study compares current visual representations, uncovering insights about training data (e.g., CLIP vs. SSL), training methods (Encoding vs. Generative), network architecture (ViT vs. ConvNext), and image resolutions. These insights can better guide developers when designing architecture, data, and methods for training native MLLMs.

We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work “bridges the gap in visual understanding”, “offers critical assessment and enhancement of MLLM benchmarks”, “introduces an innovative spatial-aware connector”, and “achieves top performance”. We summarize and respond to your questions below:

Q1: Does performance saturate with the increase of (D) and (G) in SVA

A: We recognize the importance of this point and have conducted experiments to further investigate it. We tabulate results below. We observe performance improves with increasing D & G and saturates with D > 4 or G >3.

Q2: Data, Leakage, and effectiveness

A: Our data is collected from existing open-source works. Therefore, we prevent data leakage problems by carefully choosing only the training set of data sources. For our Internet Data Engine, we focus on only Wikipedia in our project, which does not overlap with our benchmarks. In the revision, we will add a section reporting the number of test images in each benchmark present in our final Cambrian-10M dataset by checking image hashes.

Q3&4: Increasing data threshold t will not enhance the performance and whether higher data quality results in better performance

A: As we show in §5.2, data quality matters more than quantity. In Tab. 2, intermediate t values result in better performance. This result echoes observations in the data curation pipelines of CLIP and MetaCLIP Likewise, we observe that our higher-quality 7M subset results in better benchmark performance compared to training on all 10M data (Tab. 3). We believe this is a result of better balancing the dataset sources (Tab. 2, Fig. 14) and adjusting their relative sizes (Fig. 10).

Q5: response length, difficulty, diversity of instruction-tuning data

A: Thank you for raising these questions! We have conducted further analysis on Cambrian-7M and -10M regarding data composition, length of questions, length of responses, and number of rounds in the instruction tuning data and summarized the results in Tab. 1 of the rebuttal material. As a result of data curation, the Cambrian-7M data are distributed similarly to the best data ratio we found in the data ratio experiment (Fig. 10).

Q6: Compare current MLLMs using the same data

A: Like many previous studies in (M)LLMs, we argue that data is crucial in distinguishing different works. We conduct additional experiments with the LLaVA model trained using LLaMA-3-8b and Cambrian-7M data. Due to the short rebuttal period, we use the conventional 576 visual tokens of LLaVA-1.5, not the high-resolution approach of LLaVA-Next. Despite fewer tokens, this version performs comparably or better than LLaVA-Next in general, knowledge, and vision tasks. Adding our SVA further improves performance, especially in OCR & Chart tasks, while still using only 576 tokens.

Q7: Evaluate on high resolution benchmarks such as V*Bench

A: In our work, we adopt VBench in our evaluation—see “VStar” in Tabs. 4, 11, & 13. On V*Star, Cambrian-1 is competitive with LLaVA-NeXT and GPT-4V.

Q8: Determine if unfreezing visual encoder outperforms freezing in all tasks, convergence speed

Unfreezing most visual encoders outperforms freezing in most tasks. We have added a visualization to the rebuttal material (Fig. 1) that shows the %change from Frozen to Unfrozen for each model on each benchmark. Note: full Frozen & Unfrozen benchmark results are in Tables 9 & 10.

Thanks for raising the point about convergence speed. Given fixed compute, unfreezing is approximately 50-55% slower during fine-tuning. We will emphasize this drawback in the revision.

Q9: Integrate more vision encoders does not lead to higher performance on every benchmark

A: Indeed, more vision encoders does not lead to higher performance on every benchmark. We believe this is expected, as different vision encoders have different strengths/weaknesses—as studied in §3.4 and Fig. 6—and thus different combinations of vision encoders inherit combinations of these strengths/weaknesses. We will amend the 7th Finding to clarify that combining multiple vision encoders usually enhances performance, but not necessarily on every benchmark.

Q10: Provide more detail about parameter count and training hyperparameters

A: Tab. 14 provides training hyperparameters, including learning rate, batch size, weight decay, etc. We have added compute resources and training durations in the Rebuttal Tab. 2.

Q11: Could a unified vision encoder be used

A: As studied in this work, we do not have a “perfect” encoder that excels in all areas (visual grounding, language understanding, high-resolution features, etc). Therefore, we pursue hybrid encoders, which can leverage the different strengths of several pretrained visual encoders. We acknowledge that hybrid vision encoders are a work-around solution to take advantage of various pretrained models—unified vision encoders could be trained from scratch with superior performance, but that would require much more compute and data. Overall, we advocate to use MLLMs as a vision model evaluator and hope to inspire the development of a unified and powerful vision model.

Q12: More Related Works

A: We thank the reviewers for providing these new references. We will add these references to the revision. Specifically, we will add MiniGPT-4 and LLama-adapter V2 to our discussion of developing MLLMs, and we will add Visionllm, Tem-adapter, Segment and Caption Anything, and Universal Segmentation to our discussion on the downstream applications of MLLMs.