BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning

Dear reviewer,

We sincerely thank you for the detailed and constructive comments, and are glad that you acknowledge our effort in scaling data and empirical observations. Please find the response below:

W1. Incorporation of ecological or appearance descriptions

Thank you for the insightful comment. We agree that the ecological or appearance descriptions will likely facilitate the model generalization, which have been investigated at a preliminary stage of BioCLIP-XL. However, it is extremely challenging to collect such fine-grained textual information at the scale of TreeOfLife-200M that covers 952K unique taxa. Many species lack Wikipedia pages, and the descriptions often exist only in specialized taxonomic literature or original species descriptions that are not readily accessible in a structured form. Even if the species-level descriptions are available, there are inconsistencies between overall descriptions and instance images (e.g., juvenile and adult images). We view integrating richer ecological and morphological descriptions as an important direction for future work, which we expect will further improve the model’s performance and utility.

Q1. Sample distribution at each taxonomic rank

Thank you for the suggestion. Due to the rebuttal restriction regarding figures, we provide the image number distribution at different taxonomy levels in the following table. For each taxonomic rank, we report the mean, median, minimum, and maximum image number per taxon. The largest taxon at each rank is also attached following the maximum image number.

This imbalance reflects the natural prevalence and sampling biases typical in ecological data. We will include comprehensive visualizations of these distributions in the revised manuscript to better illustrate the dataset’s structure.

Q2. Balanced accuracy

Thank you for the question. While the training data is imbalanced, most of our evaluation benchmarks feature balanced class distributions.

• The Rare Species dataset has 400 classes, 395 of which have 30 unique images. The remaining 5 classes have 29, 29, 27, 27, and 21 images, respectively.
• All the test sets from Meta-Album (Plankton, Insects, Insects 2, PlantNet, Fungi, PlantVillage, and Medicinal Leaf) have balanced classes, each with 40 images.
• The Camera Trap datasets have a balanced class split. Each class in Desert Lion, ENA24, Island, Orinoquia, and OH Small Animals has 11, 56, 10, 12, 12 images, respectively.

The only exception is NABirds, where the largest class has 120 images, while the smallest one has only 13 images. We thereby report the micro (overall) and macro (class-averaged) accuracy of NABirds below. As expected, macro accuracy is lower across all models due to the imbalance. BioCLIP-XL maintains a significant performance advantage over the other models on all the tasks. We will include the test set information and comprehensive evaluation in the revised manuscript.

Dear reviewer,

We sincerely thank you for the detailed and constructive comments, and are glad that you acknowledge our effort in scaling data and empirical observations. Please find the response below:

W1. Significance of morphological-based evaluation

Thank you for the thoughtful comment. While BioCLIP-XL is intentionally focused on morphological/visual signals, we do not view the morphology-based evaluation tasks as “trivial.” Fine-grained species classification is inherently challenging due to the subtle visual differences between closely related organisms. While some biodiversity research necessitates more detailed examinations, such as X-ray or genetic sequencing, BioCLIP-XL offers the opportunity to propose hypotheses and a more targeted direction based on the morphological information.

Beyond classifying known species, we also explore the novel and more complex task of new species discovery. Using the TerraIncognita dataset [1] as an example, we evaluate the model to map the previously unseen species to the correct family based on 264K insect species names in the TreeOfLife-200M dataset as the vocabulary. BioCLIP‑XL substantially improves top‑1 family accuracy over strong vision‑language baselines:

Although assigning the exact name ultimately requires expert review and genetic examination, BioCLIP-XL offers a convenient and accurate way for hierarchical placement. This capability helps narrow the search space, prioritize comparing candidates, and accelerate the discovery. We will revise the paper to include these new experimental results.

[1] Chiranjeevi, Shivani, et al. "TerraIncognita: A Dynamic Benchmark for Species Discovery Using Frontier Models." arXiv preprint arXiv:2506.03182 (2025).

W2. Comparison with the model in BIOSCAN-5M

Thanks for the question. We agree that BIOSCAN-5M is a valuable resource and a strong prior effort in multimodal species classification. Moreover, BIOSCAN-5M and BioCLIP-XL represent two orthogonal directions to improve biological foundation models—BIOSCAN-5M introduces extra DNA information, while BioCLIP-XL studies the data scaling.

We agree and believe that the extra DNA information is critical and helpful for accurate species classification. However, as we focus on comprehensive coverage across the entire tree of life, it is challenging to acquire the DNA information for the involved 952K different taxa and 214M images. Therefore, the employment of image and taxonomic labels is based on scalability and compatibility with large-scale data. Consequently, BioCLIP-XL can be broadly applied in a variety of settings where DNA information is not easily accessible, such as historical data, citizen science images, and field observations. We will include this discussion in the revised version.

W3. Contribution of the paper

Thank you for your question. We humbly believe that “novelty” is not limited to creating new approaches or architectures. Novelty and contributions can come from demonstrating and discovering new insights. In our humble opinion, many recent breakthroughs in AI and machine learning stem from scaling up the data and learning how to train from it. With that being said, scaling up these is by no means trivial.

We also want to emphasize that TreeOfLife-200M is not a trivial aggregation of multiple data providers. As illustrated in Section 3.2 and supplementary material Section C, we design dedicated pipelines to unify taxonomic labels, filter out defective images, and remove duplicates. These efforts represent technical innovations in dataset construction and are essential to achieve a dataset of this size and the training of BioCLIP-XL.

With the scaled dataset, BioCLIP-XL achieves significant performance improvement in distinguishing species. Furthermore, this work demonstrates that BioCLIP-XL leads to broader scientific advances. We show that BioCLIP-XL serves as a general-purpose biological foundation model, providing state-of-the-art performance across a range of visual tasks beyond species classification. These properties were formally analyzed in the paper and would not be observable at smaller scales. These analyses reveal that scaling and structured supervision lead to improved and interpretable representations, providing insights for future biodiversity research and broader scientific domains.

Dear reviewer,

We sincerely thank you for the detailed and constructive comments, and are glad that you acknowledge the empirical and theoretical significance of this work. Please find the response below:

W1. Taxonomic standardization solely based on GBIF backbone

Thank you for this insightful and important critique. We fully acknowledge that relying on the GBIF backbone taxonomy as a single taxonomic reference has potential limitations. As noted by the reviewer, biological classifications are inherently dynamic, and any taxonomy could be out of date with the ongoing scientific updates. Using GBIF provides a practical, scalable approach to unify taxonomic labels across vast datasets, accepting some degree of temporal lag as a trade-off for coverage and consistency.

Built upon the unified taxonomy backbone, BioCLIP-XL illustrates capabilities beyond zero-shot classification. As illustrated in the reply to weakness 3, BioCLIP-XL learns generalizable visual features that can distinguish species and even intra-species variations without taxonomy labels. Moreover, as in the reply to question 3, BioCLIP-XL offers an opportunity to cross-validate the current taxonomic splits. Therefore, we humbly believe that the lagged taxonomy update won’t be a fatal limitation to BioCLIP-XL.

W2. Lacking spatiotemporal metadata in the dataset

Thank you for the important observation. We agree that spatiotemporal metadata are critical for many ecological and conservation applications. However, incorporating such metadata at scale presents several challenges.

• First, spatiotemporal data is often incomplete or unavailable for many images in large biodiversity datasets.
• Second, for sensitive species (e.g., endangered or threatened), sharing precise location information raises ethical concerns regarding their protection.
• Moreover, species ranges are changing. Relying strictly on expected location can limit practical utility in dynamic scenarios, such as poaching detection or invasive species detection.

With that being said, lacking spatiotemporal information during training does not limit its use for application. BioCLIP-XL is designed as a biological foundation model that can be fine-tuned to suit specific needs, including spatiotemporal context. Additionally, people can acquire a potential species list given geolocation, which has been validated in the field study to facilitate more accurate species classification.

W3. Dependency on text embedding of taxonomic names

Thank you for raising this important point. However, we would like to clarify that although BioCLIP-XL is trained with taxonomic labels, its learned visual-semantic space is not strictly bound to these specific text embeddings. Importantly, BioCLIP-XL demonstrates strong performance not only in open-set and zero-shot classification but also in few-shot settings. Given only one example image per class, BioCLIP-XL achieves 64.1% average accuracy across the 10 species classification benchmarks, outperforming its zero-shot setting (55.6%). This indicates that BioCLIP-XL does not solely rely on taxonomic labels to predict species. The visual embeddings can be used directly to construct new prototypes or support few-shot adaptation for newly described or reclassified species.

Finally, we note that BioCLIP-XL’s visual representations enable applications beyond species classification. The illustrated visual biological tasks in the paper do not depend on taxonomic labels. Thus, while taxonomic changes are an inherent challenge in biological research, we do not see them as a fundamental limitation for BioCLIP-XL.

Q1. The relationship between image-based and DNA-based methods

Thank you for this insightful question. We fully recognize that DNA barcoding is a more fundamental way to identify species. At the same time, image-based approaches such as BioCLIP-XL have distinct scientific and practical advantages that make them indispensable in many real-world scenarios.

• Scalability and data availability. Genomic data collection is often limited by cost and accessibility. In contrast, image data can be collected at scale from diverse sources, including field cameras, citizen science platforms, and historical archives. For example, our dataset comprises images of 952K unique taxa—a scale that is currently unattainable for DNA-annotated samples, especially for wild populations and rare or elusive species.
• Accessibility and non-invasiveness. Obtaining DNA samples requires specialized equipment, trained personnel, and often the physical collection or handling of organisms. However, this may not be feasible or ethical in many settings. In comparison, image-based methods allow rapid, non-invasive identification, making biodiversity monitoring more accessible to researchers and the general public.
• Complementary information to genomics. Although DNA provides ground-truth identification, it does not capture all aspects relevant to biodiversity studies. For example, life stage, behavior, health, or phenotypic adaptations are often visually apparent but not easily inferred from genetic data alone. As demonstrated in the paper, BioCLIP-XL can distinguish between juvenile and adult individuals, complementing genetic approaches.

Q2. Clarification of the incorporation of BIOSCAN-5M

Thank you for these questions. We clarify the following:

Q2.1: Labeling methodology for the images without expert morphological IDs

We used the labels as provided in BIOSCAN-5M. To ensure quality and consistency, we performed additional cleaning, including resolving residual ambiguous labels such as “Malaise####” by mapping them to higher taxonomic ranks where appropriate. This cleaning was done in close consultation with the BIOSCAN-5M dataset creators, who subsequently updated the dataset to improve label accuracy.

Q2.2: Explanation for class reduction

We are sorry that the discrepancy was not carefully explained in the manuscript. The 22,622 species in the BIOSCAN-5M paper refer to unique species-level entries. In our paper, we measure the number of unique 7-rank hierarchies instead of species. Additionally, we apply taxonomic standardization and alignment across all data providers, which resolves many irregular or placeholder labels. As a result, our resolved dataset includes 18,711 unique classes. We report in the supplementary material, line 64, that 11.8% of BIOSCAN-5M images were changed during the resolution.

We will clarify these points in the revised manuscript.

***Q3. The potential of BioCLIP-XL as a guide for taxonomy revision ***

It is an insightful comment. We believe that similarity between text embeddings of different species names in BioCLIP-XL could potentially indicate taxonomic mergers or revisions.

For example, we observe cases where the text embedding of a mislabeled species (e.g., an insect labeled as a plant) clusters more closely with the correct group (other insects) than with its incorrect label (plants). While this is a different issue from taxonomic merger, it similarly shows how BioCLIP-XL’s embedding space reflects biological and semantic relationships that can guide taxonomy-related hypotheses.

In all cases, we emphasize that these hypotheses should be validated with DNA barcoding and expert review. Nevertheless, BioCLIP-XL demonstrates the possibility to support taxonomic research by identifying potential revisions such as mergers or misclassifications.

Thank you for the detailed reply. I want to reiterate that I believe this is an amazing work from a machine learning perspective, particularly the analyses on preventing neural collapse, which is the primary reason for my positive evaluation.

However, I must clarify that my core concerns remain unaddressed. I believe these concerns stem from a fundamental discrepancy between the research focus of the machine learning community and the practical needs of biological researchers. As I see it, the primary audience for a tool like BioCLIP-XL consists of biologists seeking a robust tool for taxonomy. To that end, I would like to restate my points, which I believe would significantly increase the paper's impact and adoption within the biological community.

1. The Need for a Fine-Tuning Tutorial

For many biologists, the primary concern is scientific accuracy at the species level. While achievements in zero-shot accuracy and generalizable features are impressive from a machine learning perspective, they are secondary to the need for taxonomically precise and verifiable identifications. We cannot use labels from sources like GBIF blindly, as taxonomic accuracy is paramount.

The true value of BioCLIP-XL for our community may be as a powerful foundation model. A minimal, well-documented code tutorial demonstrating how to fine-tune the model with our own curated datasets (using updated nomenclature and verified images) would be invaluable. This would empower researchers to create downstream models that produce scientifically trustworthy results tailored to their specific needs.

2. The Scalability of Image-Based vs. DNA-Based Taxonomy for Research-Grade Identification

I would like to challenge the assertion of scalability, particularly when the goal is research-grade identification. The BIOSCAN dataset itself illustrates this point perfectly: it contains ~5 million DNA barcoded samples, but only ~460,000 with expert morphological labels required for scientific reporting. This highlights a counter-intuitive reality in biology: while images are easy to acquire, achieving a research-grade identification from them is fundamentally difficult and slow.

A research-grade species identification from images is not a simple classification. It often requires multiple viewing angles, careful examination of minute morphological features against scientific literature, and ultimately, verification by a taxonomic expert. The true bottleneck is not data acquisition, but this rigorous expert annotation process.

Conversely, while DNA can be more difficult to acquire, it provides a more direct and scalable path to a ground-truth taxonomic identification suitable for scientific studies. For large-scale research requiring high-confidence, species-level data, DNA barcoding is often the more reliable and scalable method for establishing taxonomy.

3. The Critical Importance of Spatiotemporal Metadata

I must respectfully push back on the decision to omit spatiotemporal data. The authors state that "species ranges are changing," and this is precisely the phenomenon that much of modern ecological and conservation research aims to analyze. Lacking this data severely constrains the utility of this work for biologists.

Spatiotemporal data allows us to track the spread of invasive species, analyze distribution shifts in response to climate change, and identify areas in need of conservation attention. While we all agree on the need to protect location data for sensitive species, this can be managed through established methods like data aggregation or coordinate generalization, not just outright omission. Providing large-scale spatiotemporal data would dramatically increase interest from the biological and ecological research communities.

4. Clarification on Data Distribution Across Taxonomic Ranks

Your method of mapping ambiguous BIOSCAN labels (e.g., "Malaise####") to higher taxonomic ranks implies that the vast majority of the ~4.5 million images lack species-level labels. Furthermore, your cleaning process, such as revising Orthocentrus Malaise5162 to Orthocentrus spp., actively reduces the number of species-level samples. (Note: Revision of Orthocentrus Malaise5162 to genus-only like Orthocentrus spp. is correct. However, revising it to Orthocentrus Malaise would be problematic, as the original label explicitly marks it as a distinct morphospecies.)

For the sake of transparency and to allow researchers to understand the model's training basis, it is crucial to provide a table explicitly reporting the distribution of images across the taxonomic hierarchy (i.e., number of images with confident labels at the species, genus, family, order, etc., levels). This transparency is essential for evaluating the model's capabilities and limitations.

I hope these points could help to bridge the gap between this powerful model and its intended user base.

Q4. Clarification on Data Distribution Across Taxonomic Ranks

Thank you for highlighting this important point. We apologize that this information was not included in the original manuscript.

We want to first clarify that for Orthocentrus Malaise5162 (and for similar cases of provisional labels), we omitted the species label and resolved higher ranks with the genus as Orthocentrus. It is true that the samples lack species labels. However, this sacrifice helps to make sure that an instance from BIOSCAN-5M wouldn’t be marked as unique from another instance of the same species from EOL or GBIF.

Although some ambiguous labels were mapped up to higher taxonomic ranks, our overall dataset still predominantly comprises images confidently labeled at the species level. Specifically, out of the total 214M images, approximately 92.26% retain species-level labels. Below, we provide a detailed distribution table across all taxonomic ranks:

We will explicitly incorporate this comprehensive taxonomic breakdown into the revised manuscript, ensuring full transparency of our dataset composition.

Again, we want to sincerely thank the reviewer for raising these important questions. We believe that these questions are critical for further improving the paper quality and making BioCLIP-XL a practical tool. We hope that the reply addresses your concerns. Please kindly let us know if you have any further questions.