CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale

Thank you for your review. In addition to the general response, we address your specific comments below:

First use of contrastive learning to fuse DNA and image: We thank you for pointing this out and highlighting relevant related works from the medical imaging literature which utilize human DNA. We were not aware of these works. We adjusted the language in the abstract to highlight the novelty of fusing barcode DNA and images, and we have added a subsection at the end of the related works section to discuss the works you pointed us to. While similar in their applications to DNA, we believe barcodes have a unique function as mitochondrial DNA for ecology and taxonomy compared to gene expression and other functionality for nuclear DNA, thus situating the trajectory of our work to extract information unique to barcodes.

DNA versus taxonomic labels as a target: We apologize for the confusion. Firstly, to clarify, BIOSCAN-1M has taxonomic labels for all samples, but only 3.36% of the pretraining data is labeled up to the species. For all experiments, we use all known taxonomic labels during training. That notwithstanding, we agree with the reviewer that, if all samples had species labels, and given the objective of maximizing performance at species classification, one would most likely fare better by training that model on species labels than on barcode DNA, since it is precisely our evaluation task. However, this is not the situation we find ourselves in. DNA barcodes can be acquired for a cost of less than \$1 per specimen (with costs per specimen decreasing year on year) and can be deployed at scale, whereas manual labelling costs upwards of \\$25 per specimen, and expert annotators have limited availability that can not be scaled even if funding permitted it. As a consequence of this, we are able to train our model on BIOSCAN-1M, with 1M paired samples of images and DNA barcodes, whereas a dataset of 1M insect samples labelled to species level simply does not exist. Fundamentally, this is not "a problem of the dataset" but more an issue of the practicalities of data collection and annotation at scale. Hence, conditioned on the data that is available now and for the foreseeable future, it is better to use the modality which scales better (whilst also containing information to species-level granularity), and our work and results demonstrate the success of this approach. As the reviewer points out, we are upfront that the reason DNA barcodes outperform species labels is because species labels have sparse availability. We have adjusted the text further to clarify our claims on this point, in line with this discussion with the reviewer.

Improvement at species level when aligning all three modalities: We believe that this observation is precisely one of our major claims, that cross-modal alignment can help improve alignment in the other modalities. Thus, we do believe that alignment against taxonomic labels can in certain cases benefit even image-DNA alignment. However, while all three modalities provide value, we believe that DNA is more beneficial than text as a supervisory modality for images because of its scalable collection, robustness to error and expert disagreement, and contribution to performance.

Realism of having DNA/images of unseen species as keys: Unseen species refers to species not seen during the training of the model, rather than species being discovered for the first time when passed to the model. While images and DNA may be available, retraining the model every time a new species is discovered and labelled is unlikely to be a feasible solution. It is estimated that around 80% of insect species are as yet undiscovered [1]; correspondingly, insect biodiversity surveys are constantly encountering new species, and thus it is very realistic to encounter this scenario frequently. Whilst it may be practical to retrain the model on more data periodically (after collecting multiple samples of multiple new species), we explore a solution which can operate in the intervening period, where a new species has been observed and labelled but the model has not yet been trained on it. Additionally, specimens are commonly barcoded/imaged and assigned an operational taxonomic unit (e.g. by binning) but not immediately given a scientific name. It may be preferable for the model not to be trained on these specimens until formally recognized but to still be able to match embeddings to them. In these cases, we envisage adding the embeddings of the new samples (both of new species and known species) to the vector database to use as keys for retrieval. This is the scenario we simulate by splitting out seen and unseen species, and dividing the samples into keys and queries.

[1] Stork. Ann Rev Entomology, 2018. doi:10.1146/annurev-ento-020117-043348

Methodological novelty: Please see our general response for the novelty of our work.

(cont. from previous post)

Majority of exps and evals are only shown for a single dataset: We were limited to only using BIOSCAN-1M because there aren’t other comparable insect datasets with both images and barcodes available. Where possible, we evaluated on additional datasets such as the INSECT dataset (Table 4) to demonstrate the generalizability of our method.

Aligning DNA embeddings with BioCLIP space: We thank the reviewer for their suggestion. We will update the paper with the experiment results of aligning the DNA modality with frozen BioCLIP encoders as this will enable a useful comparison on Image-to-DNA lookup against our CLIBD models. Of course, with such an experimental setup the Image-to-Image and Image-to-Text performance will remain unchanged from the results we report in Table 2, since the image and text encoders are frozen.

Reference database size: The reference database used during evaluation is the key split referred to in Figure 2, comprising 21,118 samples, each of which has an image and a barcode. We have updated the caption of Figure 2 to reflect this. We also added Table 5 to the appendix to share further statistics of the dataset, including the number of specimens with each label and the number of labels in each split.

I thank the authors for improving the writing and conducting additional experiments. I will still stay with my original rating of 3: Reject. The reasons are as follows:

1. Making claims about DNA is a better modality than the text modality just from the perspective of cost of annotation and not from an information-theoretic or computational sense is obscure, given the poor quality of taxonomic labels in BIOSCAN-1M.
2. DNA sequences can be very long, while taxonomic labels are only a few tokens. What about the computational difference (in FLOPS) between processing the DNA vs the taxonomic labels using a BERT-like model? Have you considered that cost?
3. What about the storage cost difference (in GB) between storing a database of long DNA sequences and taxonomic labels?
4. Taxonomic hierarchies are not inherently encoded in DNA and are not easier to extract. While taxonomic labels provide a more intuitive medium to understand a species type for a non-expert user.

It is estimated that around 80% of insect species are as yet undiscovered.

5. In the real world, how do you decide which species is undiscovered from the model? Some human-expert annotation is still required to verify a species not present in the training set of the model.
6. Technical novelty is still not clear.
7. The overall story and conclusions are unclear considering the points mentioned earlier.

I thank the authors for the clarifications and taking time to address each of my concerns. As I mentioned in my original review, I liked the problem motivation and completely agree that DNA barcodes can prove to be an important modality. However, the current state of the paper is such that the experimental setup and writing needs to be refined a lot to be able to make any strong conclusions. This makes the overall story blurry as the readers will be left with a lot of unanswered questions after reading the paper. As a result, i am reluctant to increase my score.

We sincerely thank the reviewer for their probing questions, as these highlighted several points upon which it would be useful to adjust the manuscript, improving its clarity and comprehensiveness.

Unimodal CLIP-style baseline: We note that the CLIP training objective is an inherently multimodal contrastive learning objective, so the methodology can not be directly transferred to unimodal experiments. Nevertheless, we agree with the reviewer that we need to have unimodal pretraining on the dataset to demonstrate that the improvement in performance comes from leveraging the multimodal training, not just adapting to the dataset domain. To address this question, we ran unimodal SSL experiments following the SimCLR methodology and found that it was still outperformed on image-image matching by the models trained with multimodal contrastive learning, achieving 10.9% macro accuracy for species compared to 51.2% by the model trained with image and DNA. We have added the full results to Tables 1 and 8 through 10 in the paper. We will also add results of training CLIBD with an image encoder pretrained on BIOSCAN-1M.

Pretrained image encoder on BIOSCAN-1M: We believe that the general visual knowledge from an ImageNet-pretrained model can still be useful but agree that starting from a model pretrained on BIOSCAN-1M could be beneficial. We will add experiment results for CLIBD using the SimCLR-trained image encoder for initialization to the upcoming revision.

Dataset confounders: While the specimens in BIOSCAN-1M are collected by different teams in three countries—Canada, South Africa, and Costa Rica—they are imaged and sequenced at one central facility, the Centre for Biodiversity Genomics (CBG). CBG employs rigorous protocols to ensure data quality. If the reviewer has more specific concerns, we would be happy to add details. We did not provide a lot of text concerning the data pipeline as it has been published in the BIOSCAN-1M paper (NeurIPS 2023).

DNA-to-DNA matching: We found that simple homology methods such as BLAST achieve higher DNA-to-DNA performance compared to contrastive learning when run on the test queries against the full key set at 99.2% macro accuracy (computed with the harmonic mean between seen and unseen), but the runtime is significantly longer (9.3 minutes to run BLAST on the unseen test samples compared to 1.1 minutes to run CLIBD and query embeddings). In addition, having dense DNA barcode embeddings can be easily incorporated into neural networks for other downstream tasks, including evolution and specialization of species in addition to species identification [1].

[1] Kress, Erickson. Methods Mol Biol. 2012;858:3-8. doi:10.1007/978-1-61779-591-6_1.

Impact of adding modalities on number of iterations: For all models, the number of samples used is the same and thus (with the same batch size used throughout the experiments), the number of training steps is the same. For the taxonomic labels, we concatenate the labels (order, family, genus, and species) up to the finest level that is known. So if the species label does not exist, we just include the labels up to the genus level. With more modalities, more aspects of the data are considered in the loss so the loss term has a larger magnitude, but this is only useful in as much as these additional contributors to the loss contain informative training signals. This is what we are seeking to investigate as the premise of these experiments.

Cross-modal matching performance trends: We account for outlier species by measuring performance by macro accuracy, with equal weight among species, rather than only micro accuracy, in which species with many samples would dominate performance. We have clarified Figure 4 to represent the proper density of species across the range of sizes and show that there are only a few outlier cases with high performances, thus having a relatively low impact on the macro accuracy. For seen species, performance increases for species with more key samples. For unseen species, image-text matching is low because the text model has not seen any of the species names, hence we do not expect good alignment for these species. For those outliers such as Xylosandrus Morigerus, in nearly all cases, the species is the most common within its genus in the dataset, thus likely resulting in better performance by extending the alignment at higher-level taxa to the lower ones. In other cases of species such as Psychoda sp. 11GMK with many samples for which the model achieves poor performance, they do not comprise a majority of the samples in the genus within the dataset or they are the only species in the genus, in which case the model has not seen the genus before either. Ultimately, this highlights a need for further improvements to cross-modal alignment and investigation into label imbalance at training and inference time, as well as understanding and exploiting further the hierarchical nature of taxonomy.

(cont. from previous post)

Cost of taxonomic label vs barcode acquisition: DNA barcoding is an automated process using specialized equipment, with the current cost below \$1 per specimen. Ongoing work is continuing to develop cheaper, less invasive DNA barcoding technology, with the goal of driving costs down to below \\$0.01 per specimen. For example, the BIOSCAN project proposed to develop protocols allowing barcode analysis for \$1 and to use this capability to analyze 10M specimens to advance species discovery. CBG achieved the \\$1 target in late 2022, allowing more than 3M specimens to be barcoded in 2023 and 2024. Metabarcoding (the barcoding of multiple specimens at once) is even more promising from a cost perspective. CBG is now targeting Oxford Nanopore Technology flow cell-based protocols that will lower the sequencing cost for barcode acquisition from \$0.05 to \\$0.001. On the other hand, obtaining taxonomic labels is currently a human driven process which can cost an average of at least \$25 per sample. Depending on whether the species is known, the detail in the image, and potential dissection involved, a taxonomic specialist may require a few minutes to often a couple hours to identify a sample. There are many specimens that are visually similar, and most specimens are not labeled to fine-grained taxa like genus or species because of ambiguities, expert disagreement, or lack of established labels.

BarcodeBERT pretraining data: BarcodeBERT is pretrained on a dataset of 1.5M barcode DNA samples from Canadian vertebrates also using the COI gene. There are no common species or barcodes between this dataset and BIOSCAN-1M, so there is no concern of data used for both BarcodeBERT pretraining and CLIBD evaluation. This pretraining dataset of barcode DNA is more similar to our data than that used for other encoders invariably pretrained on human DNA (e.g. DNABERT[-2/S], HyenaDNA), making BarcodeBERT an appropriate encoder for initialization. We have clarified the distinction in the paper and have cited the Canada 1.5M dataset.

Training with unpaired image and DNA: Thank you for your suggestion! We agree that this would be an excellent direction for future research.

Clarity of Figure 2: Please see the general response. In particular, the lines linking the numbers in the boxes indicate that the samples of the seen splits for train, val, and test have the same species, not that they are the same samples. We have clarified this figure, including the colors and the counts, in the latest revision.

Chance-level baseline in Table 1: For Image-to-DNA matching, the non-aligned results are effectively chance level. To augment this result, we added Table 7 to Section B of the appendix to show the performance of the random baseline explicitly. In addition, for the micro-averaged baseline, we also included a comparison against a simple majority-label prediction, where we select the most frequent order, family, genus, species label.

Adding geospatial modality: Thank you for your suggestion! We plan to investigate this in future work.

Thank you for your review. In addition to the general response, we address your specific comments below:

Image-to-DNA on unseen species: We do agree that the most challenging scenario for the model is in cross-modal performance on species not seen during pretraining. Despite the fact that performance is high for same-modality queries on both seen and unseen species, cross-modal queries perform much more poorly for unseen species than seen species. This demonstrates that generalization of the cross-modal alignment is the most challenging area of this task, and we anticipate future work will build on this. We believe that more taxonomically diverse data (e.g. BIOSCAN-5M, developed concurrently with this work) could help improve this performance further and will continue to investigate better multimodal embeddings.

BioCLIP comparison at higher-level taxa: We thank the reviewer for highlighting this. At all taxa evaluated, we observe that CLIBD still significantly outperforms BioCLIP for both image-to-image and image-to-text matching by 20 or more points. This demonstrates that even at taxa for which more samples have labels (e.g. order), we still see better performance by aligning with DNA barcodes. We have added Table 11 with the results of BioCLIP vs. CLIBD at higher-level taxa to Section B of the appendix.

(I-I, I-D, and I-T represent image-to-image, image-to-DNA, and image-to-text, respectively.)

Reduction in performance when adding taxonomic label modality: We discussed our analysis of this in Section 5.1 ("DNA is a better alignment target than taxonomic labels") but will improve the language of the paper to make it clearer which models we were referring to. We reason that the model trained on all three modalities performed similarly or worse in certain cases at the species level due to the low proportion of samples with species labels (~3.36% of the pretraining data) and challenges with obtaining high quality species labels, as a result of expert disagreement and visual ambiguity. Note that we use all of the samples for training, but we will only have the taxonomic labels up to the most fine-grained known level to pass to the text encoder.

Use of BIOSCAN-1M over 5M: CLIBD was developed concurrently with BIOSCAN-5M (NeurIPS 2024), which was not released at the time of our model development. We do expect the performance to improve with more data and we will further investigate the benefits of data scaling for our model.

Clarity of Figure 2: Please see the general response. Our primary goal with Figure 2 was to represent the final split statistics rather than the partitioning process. We have clarified this figure in the latest revision.

Thank you for your detailed review and your support of our work. Based on suggestions and feedback from you and other reviewers, we have revised our paper to include a discussion of the unimodal baseline (updated Table 1), experiments with ImageBind-style training (Appendix B.1.1), full taxonomic classification results in our BioCLIP comparison (Appendix B.1.2), additional statistics (Appendix A), a more in-depth discussion of DNA barcodes (Appendix D), and other information and clarifications (see general response for details).

We would appreciate it if you can look over our revised draft and our responses to your review and let us know if you still have other questions.

Thank you for your review. In addition to the general response, we address your specific comments below:

Methodological novelty: Please see our general response regarding the novelty of our work. We believe that our experimental findings can benefit downstream use cases and be extended to new problems with similar modality setups.

Comparing against non-aligned embeddings: We compare unaligned embeddings for classification against aligned models as a baseline to show the impact of multimodal contrastive learning. The reviewer is correct that in the case of image-to-DNA, as the embeddings are not aligned, the performance is effectively chance level. However, as shown in the other columns of Table 1, classification using DNA-to-DNA and image-to-image performance is fairly high even with the initial encoders, thus demonstrating improvement through cross-modal alignment. Hence we felt it was important to show the performance of the initial models on the intra-modality tasks. While we could mark the cells of this row in the cross-modal (chance) columns as “N/A”, we believe it is more informative to confirm empirically the intuition that unaligned embeddings don’t support cross-modal retrieval.

Attention visualization: Our goal with the attention visualization is to analyze how the model’s prediction changes through contrastive training, not just at the prediction level but also in terms of interpretability and feature activation. Since we are visualizing the heatmaps of the image attention outputs on the query image, the unaligned baseline is interpretable and better than chance, even if it wasn’t pretrained on insect data specifically. In the examples shown in figures 5 and 10, we observe that the heatmaps consistently focus more on the full insects for the model trained on all modalities compared to the others, supporting the benefit of cross-modal training to improve the interpretability of the image encoder. We have added additional analysis of the attention maps to Section 5.3 of the paper.

Thank you for taking the time to review our paper. Based on suggestions and feedback from you and other reviewers, we have revised our paper to include a unimodal baseline (updated Table 1), experiments with ImageBind-style training (Appendix B.1.1), full taxonomic classification results in our BioCLIP comparison (Appendix B.1.2), additional statistics (Appendix A), a more in-depth discussion of DNA barcodes (Appendix D), as well as other information and clarifications (see general response for details).

Regarding the contribution of our work, we want to emphasize that aside from “methodological novelty”, there are other ways for a work to “contribute new knowledge and sufficient value to the community” (see https://iclr.cc/Conferences/2025/ReviewerGuide), including providing empirical findings. In our work, we showed that DNA barcodes are a useful signal for aligning image representations, and we contributed a series of experiments to demonstrate this. We believe that our work can inspire more ML experts from the ICLR community to apply multimodal learning to DNA barcodes, as well as find challenging problems that may require the development of more advanced techniques. We invite you to read the position paper by Rolnick et al. presented earlier in the year at ICML [1]. This paper argues that application-driven research has been systematically under-valued in the ML community. While we do not consider our work to be solely applied, as it involves the novel application of existing ML techniques to a new problem domain and data type, we feel that a dismissal solely on the basis of “no methodological novelty” would be shortsighted and fail to recognize the substantial empirical contribution and potential impact of our work. We believe that our findings represent a significant step forward in the application of machine learning to biological data analysis and that this contribution deserves recognition within the ICLR community.

We would appreciate it if you can look over our revised draft and our responses, consider our contribution to the community, and let us know if you still have questions or concerns.

[1] Rolnick et al. “Position: Application-Driven Innovation in Machine Learning”. ICML (2024).

(cont.)

4. Improved description of the BZSL experiment on Insect dataset and its analysis (Page 10). While there were no reviewer comments specifically about this experiment, changes were made regarding concerns by R-JVaW about the use of only one dataset in our work and general writing and clarity improvements in this section.
5. (From before) Additional data statistics (Appendix A) and full taxonomic results for comparison with BioCLIP (Appendix B.1.2) in response to R-s416’s suggestion.

Other changes

1. Added discussion of related works pointed out by R-JVaW (L121-128).
2. Improved captions to clarify some figures (Figure 2,4).
3. Simplified Figure 2, and added top retrievals to Figure 11 (now expanded into Figures 11-13) to show similarity of retrieved images.
4. Other edits to motivate the use of barcodes and clarify our contribution and some methodological and experimental details (see magenta text throughout).

We hope the reviewers will consider our revised draft and update their scores accordingly. Please let us know if you still have concerns or additional questions.