Feedforward Few-shot Species Range Estimation

We thank nqXQ for their careful reading of the paper and constructive suggestions.

[nqXQ-1] Quantification of computational efficiency.
Below we report inference timings for different models (with 1 location + text), reported as the time taken in seconds to generate all evaluation species weights for LE-SINR and FS-SINR on the CPU. We observe that FS-SINR can generate the species embedding vector in as little as 2% of the time taken for LE-SINR which has to perform test time optimization to train a linear classifier for each species. In addition to not requiring any training for held-out species, FS-SINR also has fewer overall parameters compared to the SINR baseline (8.2M vs. 11.9M, see L214 Col2). We will expand on these timing results in the final revised text.

[nqXQ-2] Results as more context locations are added to Fig 7.
The quantitative evaluation in Fig 3 shows that as more context locations are added we observe an increase in agreement with the expert-derived range maps. However, we agree with nqXQ that the results do start to plateau and additional context locations provide diminishing returns (at least when averaged across all species in the IUCN and S&T evaluation sets). As requested, we report the AP for each of the species from Fig 7 below. We observe an increase in performance when increasing the number of context locations for these species.

[nqXQ-3] Uncertainty quantification.
This is a really interesting suggestion. Taking inspiration from Poggi et al. [a] we include a sparsification-based uncertainty evaluation where data is progressively removed based on the uncertainty estimates derived from an ensemble of three FS-SINR models (see Sec 4.1 in [a]). These results are on S&T using range text with different numbers of context locations. We report Sparsification Error AUC (SEAUC) and Area Under the Random Gain (AURG). AURG is positive and increases when more context locations are provided demonstrating that the ensemble is better at estimating its uncertainty than random chance and becomes more accurate as more context locations are provided.

We also visualize the predicted mean and variance for the ensemble model using the species from Fig A16. We observe high variance in locations where the individual models in the ensemble differ (e.g. South America in 2nd row). The result can be found here: https://postimg.cc/VdNTgjD0

[a] Poggi, On the uncertainty of self-supervised monocular depth estimation, CVPR 2020

[nqXQ-4] Performance of text model -- with and without images.
As discussed on L294 Col1, text describing a range is inherently more informative than images. However, in Table 1 we observe that images are still a valuable supervision source when no other meta-data is available (i.e. row 10 vs 4). Adding images with text does not hurt performance on S&T (row 11 vs 9), but does result in a drop in the more challenging IUCN dataset. This same pattern is apparent in Fig 3. The potential explanation here is that images provide sufficiently weaker signal, and greater opportunity to overfit to incorrect spurious features, thus negatively impacting performance. However, it is worth noting that even in the case of IUCN, FS-SINR with image and text still outperforms the recent state-of-the-art LE-SINR (see Fig 3).

[nqXQ-5] FS-SINR abbreviation.
Thanks for the suggestion. We will clarify this at the start of the paper.

[nqXQ-6] Plant or animal species?
We will clarify this. As noted on L241 Col1, we use the same training data as Cole et al. 2023 which contains observations for 47,375 species of plants and animals.

[nqXQ-7] Typos.
Thanks for flagging these two typos, we will fix them.

We thank HT4t for their helpful questions. By addressing these comments we believe that the description of the data processing and the evaluation protocol in the revised paper will be much clearer.

[HT4t-1] Spatial resolution of the model and data aggregation.
We use the same training and evaluation data as SINR (Cole et al. 2023), where the main difference is that by default the evaluation species are not included in our training data. As in SINR, FS-SINR uses continuous coordinates as input so there is no spatial scale explicitly defined for training, i.e. it is implicit. For the evaluation locations, we follow the same pre-processing steps as SINR, i.e. H3 cells at resolution five, which results in 2M total cells, each with an average area of ~250km^2. We will clarify this in the revised text.

[HT4t-2] Held out data comment on L260.
In Table 1 there are results for models that also train on the evaluation species (i.e. TST), but otherwise by default these evaluation species are held out of the training data.

[HT4t-3] Zero-shot experimental protocol.
We follow the same zero-shot evaluation protocol as LE-SINR (Hamilton et al. 2024). Specifically, for FS-SINR we either provide habitat (HT), range text (RT), or an image (I) for each evaluation species as input. Additionally, we also can evaluate our model when no meta-data is provided as we simply use the output of the class token as the species embedding vector (see Fig 2). In this case, the class token will be the same for all species. With the exception of the taxonomic rank text (TRT) variant which is inspired by the hierarchical species name encoding from Sastry et al., no specific species name text is provided to any of our models. We will update the relevant text to make this clearer.

[HT4t-4] Summarize ecological findings.
On L50 Col2 we note that a significant proportion of species only have a small number of location observations available. This indicates that, beyond common charismatic species, the task of species range estimation is a few-shot learning problem. We believe that the primary impact of our work relevant to ecology researchers will be better species range estimates in this few-shot regime compared to existing work (e.g. ~10% MAP compared to SINR in Fig 3 given only 10 observations). In Appendix D we also provide additional analysis that we believe will be of interest to ecologists, e.g. the spatial distribution of errors in Fig. A22, performance by continent in Fig. A23, performance versus range size in Fig. A25, and performance versus taxonomic group in Fig. A26. We will update the text to make these observations more clear and to point to relevant results in the Appendix. Thanks for the suggestion.

[HT4t-5] Hughes et al., 2021.
Thanks for flagging this paper. We will reference it in Sec 5 as it is an excellent reference related to the discussion of biases in natural world data.

We thank JzC8 for their helpful suggestions.

[JzC8-1] Performance compared to LE-SINR.
We outperform the recent LE-SINR in both the few-shot (Fig 3) and zero-shot (Table 1) settings even though we do not require any training on the evaluation species as in LE-SINR, thus making us much faster (see [nqXQ-1]). These differences can be as large as 4% MAP (e.g. row 8 vs 9 in Table 1). We will update the text on L360 Col2 to better characterize the performance improvement in a more measured way.

[JzC8-2] Value of adding images.
We discuss the image results on L294 Col1 where we say: “Perhaps unsurprisingly, in general we observe that image information is not as informative as text. This can be explained by the fact that a range text description provides much more context than an image of a previously unseen species”. Models trained on image data and no text (row 10) perform substantially better than models trained with no images or text (row 4) on both datasets (see Table 1). This shows that images are a valuable source of supervision, when no other signal is available. In practice, detailed text describing the ranges of most species does not exist, but we may have images. However, even weaker text (e.g. habitat description in row 6) is better than using images and combining images and text, at best results in the same performance (row 11 vs 9 for S&T) or slightly degrades it (row 11 vs 9 for IUCN). We will update this discussion in the revised text.

[JzC8-3] Using a different visual encoder.
As suggested, we conducted additional experiments using a different visual encoder -- a DINOv2 backbone. Perhaps unsurprisingly, this model performs worse than the features we obtain from the iNaturalist trained EVA-02 Vit (L205 Col2) in the paper. We will add these new results to the revised paper.

[JzC8-4] Results without using any text information.
Below we also report results for a model trained using only images and no text. During training the model sees between 1 to 5 images per species. Again this performs worse than when using text information, but providing images at inference helps. As noted above, text supervision is simply much more informative than images. However, using image data is still helpful when no other data is available (e.g. row 10 vs 4 in Table 1), and for most species we do not have text describing their ranges, but we can more easily obtain images.

[JzC8-5] Results using automatically generated text captions.
This is an interesting suggestion. Current vision-language models with open-weights (e.g. BLIP2) are not yet capable of generating detailed descriptions for fine-grained species images, and instead provide relatively coarse captions, e.g. “a horse standing in a field”. We generated text captions for our images using BLIP2 and evaluated an already trained FS-SINR on these captions. Here we used both no prompt and two different prompts to produce captions. “No prompt” produced captions like "a small bird is perched on a branch of a tree with flowers on it", “What species is this?” produced captions like "rufous-bellied hummingbird" and “Where is this?” produced captions like "the savannas of south africa". Results on IUCN below show that in all cases captions are worse than using the original images they are generated from.

[JzC8-6] Can the model make use of multiple images simultaneously.
Yes, this is possible. Conditioning on one image at inference time, as in the paper results in an MAP of 0.19 on IUCN (row 10 Table 1). If we instead use four images at inference, we obtain an MAP of 0.21. Thanks for the interesting suggestion.

[JzC8-7] Results in the non few-shot setting.
As requested, we conducted additional experiments beyond the 50 context locations used in the paper. Interestingly, even though this FS-SINR model was never trained on more than 20 context locations, adding more locations at inference time does not degrade performance. In the table below, shown for IUCN, we observe that performance saturates at around 50 context locations and the evaluated models gain a very small boost going from 20 to 1000 locations.

[JzC8-8] Additional suggestions, minor comments, reordering images, and typos.
Thanks for flagging these. We will address them in the revised text.

We thank QZrb for their constructive comments.

[QZrb-1] Robustness to number of training locations.
An ablation of the number of training context locations is provided in Fig. A2. We will update L250 Col1 to more clearly point to this result. As can be seen, fewer context locations (i.e. 5) performs worse, but the difference between 20 and 50 is small.

[QZrb-2] Quantifying efficiency.
Please see response [nqXQ-1].

[QZrb-3] Downstream biodiversity analysis.
Please see response [HT4t-4].

[QZrb-4] Value of images.
Please see response [JzC8-2].

[QZrb-5] Data bias.
As noted on L422 Col1, there are spatial biases in the training data we use, which has more data from North America (NA) and Europe (EU) (Fig. A24). The IUCN evaluation data is not as biased, and from the results in Fig. A23 we can see that we obtain better performance for species in NA and EU, but performance in the less sampled South America is still strong. Addressing training biases is an important question which we leave for future work.

[QZrb-6] Co-occurrence and second term in eqn 2.
This loss is borrowed from prior work (Cole et al. 2023). Without the second term in eqn. 2, the loss is trivially optimized by predicting that every species is present in every location (see Sec 3.2 in Cole et al. 2023). It is true that the second term penalizes the model for predicting that multiple species are present at the same location. However, the large value we use for $\lambda$ (see L596) means that this penalty is much weaker than the one for failing to predict that an observed species is present. Thus, if two different species occur near the same location, the model is encouraged over different batches to predict that they are both present.

[QZrb-7] Performance on out of distribution species.
To clarify, by default no data for any of the species in the evaluation datasets is observed during training (L261 Col1). By training on tens of thousands of species, FS-SINR can generalize to previously unseen species at test time. The results in Fig. A23 provide an indication of performance on less common regions. Evaluating truly out of distribution species, i.e. ones that bear no relationship to the training data, is an interesting question but would likely require new evaluation datasets. We opted to use the standard IUCN/S&T datasets so that we could compare fairly to existing work.

[QZrb-8] Performance when training on evaluation species.
In Table 1 we compare to SINR when data from the evaluation species are observed during training (i.e. TST in row 1 vs 3). It is not surprising that SINR performs better as it learns a unique embedding vector for each species (even evaluation ones), whereas FS-SINR must learn the mapping from a small number of locations to a species’ range. These results could be considered as positive as they demonstrate that FS-SINR is not overfitting by simply memorizing the text for each species. Text supervision is very sparse compared to the informative location observations used by SINR to learn its per-species encoding.

[QZrb-9] Performance wrt number of observations.
In Fig 3 we observe that for nearly all methods tested, performance improves as more observations are provided. The largest improvements are observed when going from few (e.g. 10) to many observations but begins to plateau as the number approaches 50. This is consistent with results from existing work where we see more data provides diminishing returns (e.g. Cole et al.).

[QZrb-10] Evaluation metric - recall?
We report performance using both mean average precision (MAP) (Fig 3) and a distance weighted variant of it (Fig A27). MAP is the standard metric from existing work (e.g. Cole et al. and Hamilton et al.). As a reminder, average precision is the area under the precision-recall curve, which is based on precision and recall across a range of thresholds.

[QZrb-11] Comparison to other approaches.
The Gaussian Process (GP) approach in Golding et al. is designed for presence-absence data, but we can adapt it to our presence-only setting using pseudo-negatives. We train a GP classifier using an RBF kernel and a logit link function and a Random Forest (RF) classifier. Both are implemented using sklearn using the raw coordinates as input, with a separate classifier trained per-species. We find that they do not perform as well as FS-SINR which is trained jointly on multiple species. This is especially noticeable on the more challenging IUCN dataset. The results can be found here: https://postimg.cc/0zxWzDFB

[QZrb-12] Misc.
We will improve the readability of the figures and simplify the notation.