Few-shot Species Range Estimation

We thank uR9u for their review, and respond to individual comments below.

[uR9u-1] Why 20 context locations?
In the new ablations results in Fig A1 in the revised PDF we observe that while there are slight improvements when using 50 context locations during training, there are diminishing returns compared to 20. Using 20 context locations is much faster to train and thus we use it for our main experiments. In general, we observe that after 20 context locations provided as input during inference we only see smaller improvements in the predicted ranges (see Fig 3). Regarding large range species, results broken down by range size can be found in Figs A22 to A24. There we observe a small drop in performance for very large range species, but this can partially be explained by biases in the training data as opposed to having not enough context locations (see Fig A20).

[uR9u-2] Use of pre-trained SINR model.
Importantly, the SINR encoder that we use (L252) has not observed any observations from the held-out evaluation species. As a result, it only has access to the same observation data that is available during training, i.e. this is also the same data that the baselines get. We have updated the text in Sec 4.1 to make this clear. In Fig A5 we investigate the impact of using different amounts of data to pre-train the SINR encoder, where we observe that more data improves performance as the encoder has learned a better spatial representation.

[uR9u-3] Selecting context locations at inference time.
For our results in Fig 3, the context locations are selected randomly from real observation data from those species from iNaturalist. We have provided a new detailed description of how this is performed in Appendix A. Unlike recent work such as Lange et al. NeurIPS 2023 who explore species range estimation in the context of active learning (but where they assume they have access to presence and absence data), we do not target the location selection problem. Instead, FS-SINR accepts any set in context locations as input and can make a prediction from them. Fig 3 shows that when averaged across the evaluation species, more input context locations results in better range predictions. Regarding the results for “Black and White Warbler” in Fig 4, adding more context locations is giving the model the signal that this is not just a species that has a very narrow range, but instead could be something like a bird that migrates between North and South America. There is a slight increase in the probability over India, but the scores here are much lower than those found in the Americas. Our training data contains individual species that are found on multiple continents, so predictions like this in this low data regime are not necessarily “wrong”. We also provide new qualitative comparisons in Figs A17 to A19 in the Appendix where we observe improvements as more context locations are provided.

[uR9u-4] Who is providing context locations at inference time.
As noted in the previous response, for the experiments in the paper, the context locations are randomly selected based on the real iNaturalist data (see Appendix A). Similarly, they are selected at random during training. However, at inference time these points could be selected using any approach (e.g. random, user selected, or more sophisticated selection). FS-SINR enables real-time interaction with users and we believe could serve as a valuable exploration tool for practitioners.

[uR9u-5] Post hoc few-shot learning using frozen models. Thanks for the suggestion, we will provide comparisons to such post-hoc methods in the final version of the paper. We would expect these results to be worse than the LE-SINR and SINR models which are trained during evaluation. Note, as described in Appendix A3, when training LE-SINR we regularize the learned weight vector so that it stays close to the initial zero-shot prediction which could be thought of as a kind of `class prototype’.

I thank the authors for clarifying and updating the paper with more discussion and visualizations as requested. I have updated my score accordingly. The only remaining concern is the performance of FS-SINR in the fully supervised setting (when test species are also in the training set). I believe that overfitting in this task on training species is not necessarily a bad behavior. We are essentially trying to learn a neural field over the globe, which is based on the idea of memorization. Imagine a scenario in the future where every species on the Earth has abundant observations. In such a scenario, FS-SINR could never match the performance of SINR. I really like the idea of FS-SINR being FS + SINR, which is having the benefits of SINR while also adding the benefits of few-shot mapping. This idea I don't get to see from the paper.

[uR9u-5 - follow up] Post hoc few-shot learning using frozen models.
As promised, in the new Appendix D5 (see latest revision) we present new comparisons to a new Prototypical Networks-style few-shot baseline (Snell et al. NeurIPS 2017). For the new results on IUCN in Fig A30, we use a frozen SINR or LE-SINR backbone and adapt it to the few-shot setting such that it does not require any additional training on the held-out evaluation species. The model performs much worse than our FS-SINR approach. The qualitative results in Fig A31 illustrate one such failure case whereby the Prototypical Networks' predictions get worse as the features from more locations are averaged. We would like to thank In uR9u for this suggestion as we think it is a great addition to the paper.

[uR9u-11] Performance of FS-SINR in the "fully supervised setting".
One simple way to further improve the performance of FS-SINR would be to have a learned species embedding like in LE-SINR (see Fig 2 in Hamilton et al. NeurIPS 2024). However, this comes at the cost of significantly more parameters (as noted in Sec 4.1, FS-SINR has half the number of learnable parameters as SINR). We should also emphasize that it is only in the zero-shot setting, i.e. not the few-shot setting as in Fig 3, where FS-SINR performs `worse' than the SINR variant that has trained on the evaluation species (see Table 1 - row 1 vs row 3). This is because of the relatively weak signal provided by the input text. However, we can see that our FS-SINR approach, not trained on the evaluation species, obtains relatively similar performance (Fig 3 / Table A3/A4; 0.67 IUCN and 0.71 S&T) to the SINR model that has been trained on the evaluation species (Table 1; 0.67 IUCN and 0.77 S&T for SINR) with only 20 samples and no text.

We thank NdXm for their feedback, and respond to individual comments below.

[NdXm-1] No significant theoretical/technical contributions.
We present a new approach for few-shot species range estimation and we show that it outperforms existing state-of-the-art methods (see Fig 3). Our main technical contribution is a novel architecture for this task that enables few-shot estimation in a single forward pass (Fig 2) without requiring any retraining. Existing methods such as SINR (Cole et al. ICML 2023) and LE-SINR (Hamilton et al. NeurIPS 2024) can only be applied to the few-shot setting with additional retraining. Species range estimation is an important problem, and we believe that the few-shot setting will significantly benefit from input from the machine learning community.

[NdXm-2] Dataset biases.
As noted on L426, like several previous neural-network methods (e.g. Cole et al. ICML 2023, Lange et al. NeurIPS 2023, Sastry et al. arxiv 2023. Hamilton et al. NeurIPS 2024, …) we do not explicitly account for spatial biases in the data. There have been some attempts in the machine learning literature to address this (e.g. Chen et al. AAAI 2019). We agree that this is indeed an important consideration, we believe that it is a complementary research direction to the one explored in our work which is focused on few-shot estimation. Future advances in loss functions and de-biasing techniques could be combined with our model.

[NdXm-3] Uncertainty modeling and the difference between 1 and 5 context locations.
In the results in Fig 3 we observe large differences in MAP when comparing models that have 1 versus 5 context locations. For example, for “FS-SINR (No Eval Text)”, there is a difference of ~0.15 MAP on the IUCN data. While they may appear small at a global scale, in Fig 4 we do see noticeable differences between the model predictions when the number of context locations is increased. Fig 5 also demonstrates that large differences in the output predictions can be observed when changing the input text. This demonstrates that our model is responsive to user guidance. Additional new results in Figs A17 to A19 also show how the predicted ranges become more refined as more context locations are provided. However, we also agree with the suggestion that it would be beneficial to have stochasticity in the model predictions. As noted on L420, this could be achieved via an additional sampling layer. To demonstrate that FS-SINR does produce varying outputs, we present new results in Fig A15 where we show that different random initializations (i.e. different random seeds) during training result in very different predictions in the zero-shot setting.

[NdXm-4] Related work on few-shot learning.
L108-L123 discussed related work on few-shot learning in the context of species range estimation. There are many aspects of the problem that make it distinct from few-shot learning explored in other domains (e.g. in image classification). For one, the input domain is fixed (i.e. all the locations on earth), each location can support more than one species (i.e. multi-label as opposed to multi-class), the label space is much larger (i.e. tens of thousands of species as opposed to hundreds of classes in image classification), and only partial supervision is available (i.e. we only have presence data, with no confirmed absences). We agree that there could be many exciting connections to the larger literature on few-shot learning and hope that our work, and benchmarking, motivates further research in this direction. To address your comment, we have updated the related work section to make these distinctions clearer. Thanks for the suggestion!

[NdXm-5] Additional qualitative results.
We have provided a new section of additional qualitative and quantitative results in Appendices C and D respectively. There, we demonstrate additional qualitative comparisons to LE-SINR and SINR (e.g. see Figs A17 to A19) and LD-SDM-like taxonomic text in Fig A9.

[NdXm-6 Q1] MAP ignores geographic distance?
This is a good suggestion. For fair comparison to existing work, we use MAP as it is the standard metric used by the recent baselines we compare to (e.g. SINR, LE-SINR). LD-SDM (Sastry et al. arxiv 2023) proposes an alternative metric which takes the distance from the nearest observations into account. In their work, it is applied in the context of presence-only evaluation data. There are some design decisions that are important to take into account if one was to adopt their metric. For example, should one measure distance from a test location to the closet, or average, presence location. We will implement a weighted MAP metric and report results in the final version of the paper as it is not possible now due to time constraints. Based on the qualitative results already in the paper, we do not expect that this metric will result in any differences between the rankings of methods.

[NdXm-4 - follow up] Related work on few-shot learning.
In Appendix D5 we present new comparisons to a new Prototypical Networks-style few-shot baseline (Snell et al. NeurIPS 2017). This was suggested by uR9u. For the new results on IUCN in Fig A30, we use a frozen SINR backbone and adapt it to the few-shot setting such that it does not require any additional training on the held-out evaluation species. The model performs much worse than our FS-SINR approach. The qualitative results in Fig A31 illustrate one such failure case whereby the Prototypical Networks' predictions get worse as the features from more locations are averaged.

[NdXm-6 Q1 - follow up] MAP ignores geographic distance.
As promised, we implement the suggested distance weighted evaluation metric. This was something that vJfa also wanted to see based on NdXm's review. We think this is a great addition to the paper and we thank NdXm for suggesting it. As predicted the new results on the IUCN dataset in Fig A28 demonstrate that the rankings of different methods are not impacted by the weighting. However, overall performance does decrease for all methods. We describe the new distance weighted metric in Appendix D4 and illustrate qualitative examples in Fig A29.

Additional comments about the figures:

[vJfa-7] Qualitative results for Hyacinth Macaw in Fig 4.
In the case of the Hyacinth Macaw (right column) in Fig 4, there is strong spatial bias in the data whereby the overwhelming majority of observations for the species come from the south west of Brazil, while the species has a larger range encompassing remote locations. This can be seen clicking on the “Map” tab on iNaturalist for this species:
https://www.inaturalist.org/taxa/18938-Anodorhynchus-hyacinthinus

As a result, random sampling of locations for this species results in very similar spatial locations being returned. See our response [uR9u-4] for more discussion of sample selection.

[vJfa-8] Figure clarity in Appendix.
We have significantly updated the Appendix such that the results are better organized and easier to follow.

[vJfa-9] Report the results from Fig 3 in a table in the Appendix.
Good idea. We have added these results to Tables A3 and A4 in the Appendix.

[vJfa-10] Add the text descriptions for Figs A7 and A8.
We have added the habitat and range text for these species in the figures (now A13 and A14).

[vJfa-11] Error bars for the ablations in the Appendix.
As requested, we reran and updated the ablations and added error bars. For clarity, we present results with and without error bars so that it is easier to see the results of different methods.

We thank vJfa for their detailed comments and feedback. We respond to individual comments below.

[vJfa-1] Limited technical contribution.
We present a new approach for few-shot species range estimation and we show that it outperforms existing state-of-the-art methods (see Fig 3). Our main technical contribution is a novel architecture for this task that enables few-shot estimation in a single forward pass (Fig 2) without requiring any retraining. Existing methods such as SINR (Cole et al. ICML 2023) and LE-SINR (Hamilton et al. NeurIPS 2024) can only be applied to the few-shot setting with additional retraining. Species range estimation is an important problem, and we believe that the few-shot setting will significantly benefit from input from the machine learning community.

[vJfa-2] Provide more ecological analysis.
Thanks for this suggestion! Based on this request, we have added much more analysis to the Appendix to provide a deeper understanding of the performance of our method. In Fig A16 we visualize the learned spatial representation which demonstrates that geographic features emerge during training like in SINR. In Fig A20 we visualize false positives on the map where we see more error in Africa and South America compared to Europe and the US. This is likely explained by spatial biases in the data. In Fig A21 we visualize the same types of errors for models that either use range, habitat, or no text. In Figs A22 to A24 we report performance per-species binned by their range size, where we observe that small and very large range species are challenging. Finally, in Figs A25 to A27 we report performance across coarse taxonomic groups from the evaluation set (i.e. amphibians, birds, mammals, and reptiles), where we observe that there are no major differences in performance across different types of species. We hope that this additional analysis will make our work more appealing to ecologists who might use it.

[vJfa-3] Model output is deterministic.
The current model is deterministic, but as we see in the example in Fig A11 adding context locations (e.g. first in North Africa, then in South America, and finally Australia) significantly alters FS-SINR’s predictions. As noted on L420, this could be achieved via an additional sampling layer. To demonstrate that FS-SINR does produce varying outputs, we present new results in Fig A15 where we show that different random initializations (i.e. different random seeds) during training result in very different predictions in the zero-shot setting.

[vJfa-4] How are the error bars generated?
We have updated the description in the implementation details in Appendix A to clarify this point. Specifically, we perform three runs for each experiment using different seeds and report the mean. We display the standard deviation as error bars in our figures.

[vJfa-4] How does performance depend on the type of habitat or range size?
Please see the response to query [vJfa-2], where we point to new results that break down performance based on geographical location (Fig A20) and range size (A22 to A24)

[vJfa-5] Incorporating coarser taxonomic information.
Based on this suggestion we added new comparisons to the type of taxonomic rank text used in LD-SDM (Sastry et al. arxiv 2023). Instead of using free-form text as input, this involves using a text string that encodes the taxonomic hierarchy of the species (i.e. species, genus, …). New zero-shot results are provided in Table 1 in the main paper with a full breakdown per-rank level in Table A2. More details and visualizations (e.g. Fig A9) of these experiments are provided in Appendix B7.

[vJfa-6] Has the language encoder learned taxonomic hierarchy.
In Fig A8 we observe that the model that uses Wikipedia-derived free-form text outperforms all forms of taxonomic text, with range text being superior. However, as the number of content locations increases the performance difference is smaller. While this does not establish if the language encoder represents taxonomic hierarchy, it shows that free-form text results in superior range predictions in the few-shot setting.

We thank EQN8 for their detailed comments and feedback. We respond to comments individually below. One potential misconception that we would like to clarify that is related to more than one comment is that in our few-shot setting the species in the evaluation set are disjoint from those in the train set (L84 and L269 in the original submission), i.e. our evaluation species are not part of the training set during model training.

[EQN8-1] Novelty wrt LE-SINR (Hamilton et al. NeurIPS 2024).
The main similarity between our FS-SINR approach and the recent LE-SINR (Hamilton et al. NeurIPS 2024) is that both can use text data to assist with species range estimation. However, LE-SINR is not the first method to text, e.g. LD-SDM (Sastry et al., 2023 arXiv 2023) also uses text, albeit in a more restricted form. Otherwise, our setting is very different from LE-SINR. Specifically, we perform few-shot species range estimation without any retraining, whereas they are addressing the zero-shot setting. Their method cannot directly incorporate location observations without retraining a linear classifier. On L118 of the original submission we note the main distinction to LE-SINR where we say that “LE-SINR performed few-shot experiments whereby they used a language encoder to estimate an initial encoding for a species and combined it with a linear classifier that needs to be trained to generate range predictions.“ On L84 we note that our main contribution is that we can “predict the spatial range of a previously unseen species at inference time without requiring any retraining.” Our results in Fig 3, demonstrate a significant improvement over their method, where we observe that our predicted ranges are quantitatively superior, while also not requiring any retraining. We also see in the new qualitative examples in Appendix C2 (i.e. Figs A17, A18, and A19) especially in the low observation setting, that show that our outputs are qualitatively better. As suggested in the reviewer comment, we have updated the text in the related work section so that this distinction is much clearer. Thanks for the suggestion.

[EQN8-2] Ablations to demonstrate which of the components contribute the most to the improved results.
We have added a much more detailed set of ablation experiments in Appendix B to justify the main modeling components used in FS-SINR. There we explore the impact of multiple factors such as the amount of training data, input features, architecture choices, number of context locations, and different combinations of context information.

[EQN8-3] Provide a learnable species token as input to FS-SINR.
One of the main advantages of our FS-SINR approach is that it has significantly fewer parameters compared to SINR. In total, our model has 6.3M learnable parameters compared to 11.9M for SINR. Note, in the original submission (L255) we incorrectly stated this was 29M due to an error in how we counted the parameters in PyTorch. The challenge with the suggestion of providing a learnable species token for each of the training species is that it will then not be possible to evaluate held-out species that are not observed during training. As noted on L269, we hold out the evaluation species from the train set and thus no observations from them are observed during training. Please do not hesitate to let us know if we have misunderstood your suggestion.

[EQN8-4] Adding more than 50 context locations.
The most amount of context locations we evaluate with is 50. We justify this choice because we are operating in the few-shot setting. If more than 50 observations were available for a species, it would make more sense to add them to the training set. While the performance increase does plateau after a set number of observations (see Fig 3), interestingly, we observe that our approach performs very close, and sometimes even better, compared to the SINR baseline that has seen observations from the test species during training (i.e. first row in Table 1). For example, the SINR model obtains MAP of 0.67 on the challenging IUCN dataset, whereas our FS-SINR approach obtains a similar score with only 20 observations (see Fig 3 - left). Importantly, FS-SINR only requires a single forward pass through the model to obtain this score (L274). In the updated paper we provided additional results where we ablated the impact of different numbers of context locations at training time (see Appendix B.1).

We thank the reviewers for engaging in the discussion and for their valuable comments which have contributed to improving the paper greatly. Based on the suggestions we provide a final set of requested comparisons. These can be found in the revised PDF.

New evaluation metric.
In Appendix D4 we present new results using a distance weighted variant of the main evaluation metric we used. This was in response to a request from vJfa and NdXm. From the results in Fig A28 we observe that the relative ordering of the different methods does not change when using this new metric, and our FS-SINR approach still performs best.

New few-shot comparisons.
In Appendix D5 we present additional comparisons to a Prototypical Network-style post-hoc baseline. This was requested by uR9u. The results in Fig A30 demonstrates that FS-SINR outperforms it.