Combining Observational Data and Language for Species Range Estimation

[CYPb-1] The model is given range/habitat information as input.
It is true that the model is given this information as free-form input text. However, these range descriptions do not provide enough detail to draw a precise range map. For instance, knowing that the Gray Kingbird breeds in the "extreme southeast of the United States, mainly in Florida" (Appendix B, Example #1) is not enough information to draw a map of its range within the United States. What parts of Florida? And where in the southeastern U.S. outside Florida?

We demonstrate that it is possible for a model to make use of this relatively coarse text information to produce strong zero-shot and few-shot performance (see Fig. 3). This type of range information could indeed be readily available for a species that was historically recorded to science with no spatial observations or simply for species with very few observations (i.e. the vast majority of species on platforms such as iNaturalist only have a few observations and thus would benefit from better few-shot range estimation models).

[CYPb-2] Results when training with text only L145-L147.
We present the results for our model without the learned species tokens (“LE-SINR no species tokens”) for the zero-shot setting in the table below. Its performance is similar to our full model where these tokens are learned (“LE-SINR”). The advantage of including the learned tokens is that we can evaluate on those species directly if they are included in the training set. We will update the text on L145 to more precisely quantify their performance similarities in the zero-shot setting.

[CYPb-3] Adding evaluation species to the training set.
As requested, we performed an additional experiment where we added the evaluation species to the training set (“LE-SINR w. eval species”) and compared these results to our model from the paper where they are excluded (“LE-SINR wo. eval species”). Perhaps unsurprisingly, adding the evaluation species improves performance. We will add these results to Table 1 in the revised text. Note, our performance is lower than the SINR baseline in Table 1 as we are only using LLM summarized text as input at evaluation time (L206) which differs from the training text (L186), whereas SINR gets to use the learned species tokens.

[CYPb-4] Comparison to different location encoders.
Our approach is agnostic to the choice of location encoder used. As requested, we performed additional experiments where we compare to GeoCLIP (Cepeda et al. NeurIPS 2023) and Spherical Harmonics (Rußwurm et al. ICLR 2024). In both cases, these encoders also make use of our LLM encoder. In the table below we observe that our LE-SINR approach outperforms these approaches. These results are perhaps not too surprising, because as noted in Table 1 (c) in Rußwurm et al. ICLR 2024, their spherical harmonic encoding does not actually perform better at the geo prior task (the one most closely related to range estimation task) compared to standard “wrapped” encoding that we use. Additionally, for the GeoCLIP experiment, where we start with their pre-trained network instead of our location encoder and fine-tune it, we also outperform them. This is also not surprising as their encoder is trained on web-sourced images that depict common everyday categories and are not necessarily specific to the natural world.

[CYPb-5] Discussion of what the text encoder has learned.
This is a good suggestion, we will include additional discussion regarding what our language encoder has learned. Regarding hierarchy, our comparisons to LD-SDM (see response to 9uUS) indicate that simply encoding taxonomic hierarchy results in sub-optimal performance. In response to q7Tg’s question we also plan to illustrate which parts of the text are important to the model.

[CYPb-6] Use of the words “location” and “position”.
Thanks for flagging this, we will make our usage of the terminology more consistent.

[CYPb-7] Species branch (L139-L143).
The species branch refers to the two mechanisms we have for encoding species information: our text-based $g()$ encoder and learned species tokens $E$. We will refine this text and Fig 2 to make this clearer.

[CYPb-8] Clarification of loss function computation (L148-L152).
We will add the suggested text to improve the description in this section.

[CYPb-9] Description of training loss in Section 3.4. We will update the text in this section to make it easier to read and add an equation such that the comparison to SINR is clearer. Thanks for the suggestion!

[9uUS-1] Comparison to LD-SDM.
While related, LD-SDM (Sastry et al. arXiv 2023) uses text data in a fundamentally different way to us. Instead of using unstructured text as input, their model generates an input string for each species that encodes its full taxonomic hierarchy (e.g. class -> order -> family -> genus -> species). Then for their zero-shot evaluation, they simply remove the final species name from the input text and evaluate on previously unseen species. Their core assumption is that they have seen species from that genus at training time. Our approach does not make such a strong assumption.

As requested, we performed an additional comparison to LD-SDM. We use the same architecture as our LE-SINR method with environmental feature inputs and simply change the training text. As our text encoder does not use causal attention, during training we randomly blank out different tokens to simulate the full taxonomic hierarchy not being present in the text string at evaluation time. In the table below, where the evaluation species are part of the training set, we crop the input text to different taxonomic levels during evaluation to simulate having coarser information (e.g. the “genus” row, corresponds to only including the full taxonomic name up until genus at test time).

In comparison, our approach performs significantly better.

[9uUS-2] Comparisons to contrastive losses.
As requested, we perform additional comparisons using a contrastive training loss, specifically the one from SatCLIP (Klemmer et al. arXiv 2023). During training we compute a contrastive loss between species and locations within a batch. We also extend this method (SatCLIP + negs.) by adding 50% uniform random locations as additional negatives. However, in both cases we see that it underperforms compared to our method. Additionally, our LE-SINR approach is compatible with different input encodings. See response to CYPb for further comparisons.

[9uUS-3] Equations 1 and 2.
Yes there is a minor typo here, thanks for flagging this. We will remove the extra minus at the start.

[9uUS-4] Which parts of the model are frozen for few-shot evaluation.
During few-shot evaluation, the entire model is frozen (i.e. the location encoder $f(\mathbf{x})$ and the text-based species encoder $g(\mathbf{t})$). We only optimize the final classification weight vector for the species of interest. We will update Section 3.6 to clarify this.

[9uUS-5] X-axis in Fig. 3 (left).
As indicated in the caption, this figure displays both zero-shot (i.e. “Observations per Species” = 0) and few-shot (i.e. “Observations per Species” > 0) performance. Only our language enhanced models are capable of producing zero-shot predictions. We will update the caption to make this clearer.

[q7Tg-1] Which parts of the text were most informative for the LLM?
This is a great suggestion. Given the space limitations we cannot provide a visualization of this in the rebuttal, but we will include this in the final revised paper. At a high level, text which more directly encodes information about the range is more informative than text that just describes the habitat (see results in Fig. 3).

[q7Tg-2] Does the model work for species that are common to more than one location?
Just like SINR, there is nothing conceptually stopping the model from predicting that a species occurs in more than one location on earth. In Fig. B3 in the rebuttal PDF we illustrate two examples of text that describe different climate zones that overlap with multiple different countries. We will include some additional species’ examples in the final revised text.

[q7Tg-3] For zero-shot evaluation, can the model generalize beyond locations explicitly mentioned in the input text?
Yes. The example in Fig. B3 in the rebuttal PDF shows examples where specific locations are not mentioned, yet the model produces sensible predictions. Some of the examples in Fig. 5 in the main paper also demonstrate that the model is able to spatially encode non-species concepts from a small number of non-geographic words (e.g. “Hello Kitty” or “Babe Ruth”). We believe that exploring how LLMs spatially represent concepts is an exciting future research direction.

[q7Tg-4] Can we use additional ecological concepts as side-information?
This is a really interesting suggestion! Beyond the scope of this work, but it could be possible to use text related to species interactions (or lack thereof) to further improve the spatial representations we learn. Such knowledge is not likely explicitly codified at a large scale in a structured way online, but this makes natural text an ideal candidate to encode it.

[q7Tg-5] Can the model infer species’ presence in similar habitats?
Yes. We believe that the model is already doing reasoning of this form. However, there are some possible limitations of non-spatially explicit encodings (i.e. only using environmental features and not spatial coordinates as input). When only using information derived from environmental habitats, it could be possible for a model to incorrectly predict that a species is present in that habitat across different continents (see “desert” example in bottom row of Fig. 5).

[q7Tg-6] Can we estimate population size instead of just presence and absence?
This is a longstanding and open question in statistical ecology, and out of scope for this initial work. This question has been studied extensively in the literature (e.g. Pearce and Boyce, Journal of Applied Ecology 2005), but without information about true absences or individuals it remains under constrained. However, proxies such as relative abundance can sometimes be estimated.

[q7Tg-7] Estimating the conservation statuses of different species?
As noted in Section 4.4, great caution should be exercised when using the outputs of our work for any downstream conservation assessments. While species range is an important factor in determining the threatened status of a species for repositories such as the IUCN Red List, other information is also needed (e.g. the list of threats they are susceptible to).

[A4uw-1] Within scope for NeurIPS?
We believe that NeurIPS is an appropriate venue for this work given similar work published at top ranked machine learning venues in the past. For example, NeurIPS (“Active Learning-Based Species Range Estimation” by Lange et al. and “SatBird: Bird Species Distribution Modeling with Remote Sensing and Citizen Science Data” by Teng et. al.), ICML (“A Maximum Entropy Approach to Species Distribution Modeling” by Phillips et al. and “Spatial Implicit Neural Representations for Global-Scale Species Mapping” by Cole et. al.), ICLR (“Geographic location encoding with spherical harmonics and sinusoidal representation networks” by Rußwurm et. al.), and AAAI (“Bias reduction via end-to-end shift learning: Application to citizen science” by Chen and Gomes). We believe that new methods for estimating the spatial range of species is a question of societal importance and requires input from machine learning researchers. Additionally, insights into how spatial information is encoded into LLMs is highly relevant to the machine learning audience.

[A4uw-2] Moving equation 2 for enhanced readability.
Thank you for the suggestion! We will make this fix.

[A4uw-3] Köppen climate zone descriptions.
Evaluating more species related text descriptions such as Köppen climate zones is an interesting suggestion. As requested, we have included two zones in Fig. B3 in the rebuttal PDF. We observe LE-SINR is able to give plausible estimates of these zones, particularly where iNaturalist has reasonable training data coverage. However we observe false negative predictions in areas of low data coverage (e.g. central Africa in the figure on the right).

[2RN9-1] Motivation and use case.
The primary motivation of our work is to leverage an additional data modality, text, to improve both zero-shot and few-shot species range mapping. We observed that text data, as formatted on Wikipedia, often includes descriptions, habitat information, and range information for species. We acknowledge that the presence of detailed range descriptions can simplify, and in some cases trivialize, the process of producing a range map for a species. This is precisely why we also conducted experiments using only habitat information. While performance decreases compared to using range descriptions, we demonstrate that incorporating habitat text still offers improvements over previous methods (see Fig. 3).

As a motivating use case, we envisioned a scenario where a scientist, recently returned from the field, describes the habitat in which they observed a possibly new species and uses our method to visualize a plausible range map. In practical applications, the maps generated by our method are intended to serve as a starting point for further refinement. This initial starting point can be valuable for generating detailed maps and guiding more in-depth studies. A large percentage of species on platforms such as iNaturalist only have a small number of observations, but do have Wikipedia pages, and thus would benefit from better few-shot range estimation. Furthermore, some species are presumed extinct, have no modern observations, but still have text descriptions on Wikipedia (e.g. “New Caledonian owlet-nightjar”).

We understand that the use of habitat text could conceivably create output maps that show the “fundamental ecological niche” of the species rather than the range or the similar “realized ecological niche” of the species. In practice, as noted in the review, our model does manage to “restrict” geographically. This is not due to post processing. Instead our model is able to infer the range from the habitat text due to mentions of other species, and specific features of that location. For example, for the hyacinth macaw, the habitat text ends with: "these parrots are found... in dry thorn forests known as caatinga, and in palm stands or swamps, particularly the moriche palm (Mauritia flexuosa)." World knowledge encoded in the LLM about "caatinga" and "Mauritia flexuosa" allows the model to successfully select from the many locations that fit the first line of the description of "semi-open, somewhat wooded habitats" and so correctly chooses South America as the likely home of this species.

[2RN9-2] Baselines.
The Model Mean model is the Oracle SINR model (trained with or without environment features), but whose output is the average of all species outputs (including or excluding the evaluation species) for each input. Model Mean +Env +Eval Sp. is therefore the Oracle SINR model trained with additional environmental features as input and is trained with observations from the evaluation species. We will clarify this in the text, sorry for the confusion!

In response to requests from other reviewers, we have included additional quantitative comparisons, e.g. LD-SDM (Sastry et al. arXiv 2023) and the contrastive SatCLIP (Klemmer et al. arXiv 2023) in response to 9uUS, GeoCLIP (Cepeda et al. NeurIPS 2023) and Spherical Harmonics (Rußwurm et al. ICLR 2024) in response to CYPb. See the responses to the other reviewers for results.

[2RN9-3] Results.
Our method is advantageous in both the zero-shot and the low-shot setting, where we are much better than the existing SINR. The results in the low-shot setting are particularly noteworthy, as we observe a consistent performance improvement all the way up to, and including, the 10 training observations per species setting. This is important as thousands of species on platforms such as iNaturalist have a limited number of observations.

We agree with your desire to control for the negative sampling process in our few-shot results. We provide these additional results in Fig. B1 in the rebuttal PDF. We observe the relative ordering of the different methods stays the same, and we still observe a large boost in performance from our method compared to SINR.

[2RN9-4] Error bars.
The results in Fig. 3 are averaged over hundreds of species, and as noted on L617 the standard deviation is very small. We can include them in the final revision if deemed important.

[2RN9-5] Species in evaluation set with no text descriptions.
For training species with no text descriptions we only use the learned species tokens, i.e. SINR approach. For evaluation species with no text description we just skip them and set the performance as 0. There are four evaluation species with no Wikipedia text.

[2RN9-6] Analysis of whether certain regions are better predicted.
Interesting suggestion! Please see Fig. B2 in the rebuttal PDF for an investigation into how our performance is biased geographically. We observe that, perhaps unsurprisingly, our approach underperforms in regions with limited training data (e.g. central Africa), with a particularly high error for Lake Victoria. As there are very few training examples in this region our model has almost no understanding that this lake exists and gives similar predictions for the lake and surrounding land areas, despite the large difference in species present on land and in water.