BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics

We thank Reviwer sjxb for their feedback. We hope the following addresses the weaknesses and questions raised in their review.

The paper claimed the finding of "increasing the size of networks does not translate to improved generalization". However, no reason for this phenomenon is given.

This is a good question. Prior works have also made similar observations (for instance "A Unified Few-Shot Classification Benchmark to Compare Transfer and Meta Learning Approaches"). Further investigation is necessary, but our intuition is that either a) scaling up capacity is not an effective strategy in challenging transfer tasks, or b) due to class imbalance, we are in a medium-to-low data regime for a significant proportion of the species that is not supportive of larger network architectures.

This finding highlights the importance of our objective of designing a benchmark that is informed by a realistic problem setting. Standard academic benchmarks are akin to unit tests for deep learning approaches: methods developed using those benchmarks can and do overlook important failure cases in real-world applications, which can only be uncovered through "integration test" benchmarks like BIRB.

We will incorporate this discussion in the updated manuscript.

Does each of these datasets emphasize something (e.g. Label shift influence, long-tail distribution)? If not, they lack of research potential to be set as a part of benchmark dataset.

Each of the datasets maps to one of the tasks we describe in Section 4.3, each of which aligns to one or more challenges that we measure by evaluating on that dataset. While the alignment is not necessarily one-to-one, the datasets are diversified across geographical locations and species distributions, which helps establish generalizability in the benchmark results. We also note that the "Disentangling Label and Covariate Shift" subsection in Section 4.3 does present ablations that directly disentangles the various learning challenges presented by BIRB. The methodology we present there can be followed by users of the benchmark to study the effect of those challenges in isolation.

"Integration" benchmarks like BIRB are important to deep learning researchers for the reasons outlined in our response to Weakness 1. Our benchmark is intended to drive fundamental research in a way that is directly beneficial to ecological researchers and conservationists, whose modelling challenges do not neatly decompose into different research subfields. Transfer learning, domain adaptation/generalization, and class-imbalanced learning applications all have the opportunity to demonstrate effectiveness on BIRB, and we believe that the fact that such advances can directly translate to real-world impact will be encouraging to many researchers in those fields.

Throughout the paper, no example dataset sample is provided to give intuition of how the dataset is constructed.

To clarify, are you referring to an example as it goes through the data processing pipeline, or do you mean more explicitly the modality/format of an example? We'd be glad to provide a more explicit description and/or additional modality to illustrate this more clearly--please let us know which component you were referring to.

Regarding the two questions raised in the review:

Is it a reasonable practice to use existent public datasets to construct one's own dataset? Are the credits properly given?

There is precedent in constructing "datasets of datasets" benchmarks; refer for instance to "A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark" or "Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples". We agree that proper credit attribution is important in those cases, which we do through citations in Table 1. We will clarify in the text that we are making use of data collected as part of previous works to help avoid any ambiguity, and we are happy to incorporate any further suggestion on how to make that clear.

We also note that the datasets we employ are designed for ecologically themed research and are not usable off-the-shelf in machine learning applications. They do not prescribe any specific evaluation procedure; the evaluation protocol we propose is our own contribution. The datasets were also prepared by a variety of different groups, which required significant effort on our part to achieve data consistency across datasets. Preprocessing details (for reproducibility) are outlined in Appendix 1, which we summarize here:

• Resolving significant differences in the bird taxonomies used by various datasets
• Correctly and consistently aligning labels in various input formats
• Extracting fixed length slices from file-level labels/annotations using peak-finding (Xeno-Canto queries)
• Converting timeboxed annotations in soundscape data into fixed length labeled slices

What does BIRB stand for? Apologies in advance if I missed it in the paper.

Thank you for pointing this out. We will clarify by expanding the acronym (Benchmark for Information Retrieval in Bioacoustics) the first time it is used in the main text. We will also add a footnote to an online article (https://www.audubon.org/news/when-bird-birb-extremely-important-guide) to provide context on the origin of the word.

We thank Reviewer sG4k for their feedback! We address some concerns and include follow-up questions so we can better understand the concerns.

Similarly, the benchmark itself is also unclear. The task should be better to illustrate with a figure and the figure of baseline system is not informative.

Would you be able to clarify the aspects which are unclear so we can address these either with clarifications in our response or improve the presentation of the paper? We care about accessibility of this system and reader understanding!

Some relevant information in supplementary may be moved into the main draft to elaborate the benchmark. In addition, for the experimental analysis, corresponding discussion and visualizations are necessary for a better description.

Are there any pieces included in the supplementary that you found particularly helpful to highlight in the main paper? For the sake of providing sufficient introduction to the setting, the dataset descriptions, benchmark setup and protocol details, suite of evaluation results and analyses, etc., there is a lot of content and discussion to cover and we’re trying to strike a balance between providing enough information in the main paper and opportunities for the reader to learn more by referencing the Appendix where we elaborate on all of these areas.

Given the current machine learning field not familiar with the bird-based bioacustics, more background information should be included before introducing the benchmark.

Similarly to the previous point, if you have any specific suggestions on what you found or would find helpful, please do let us know!

We thank Reviewer N4MM for their thoughtful feedback and questions! We address these concerns and questions.

[The benchmark] could benefit from more innovative approaches or methodologies for retrieval tasks. I think it's important to inspire researchers with novel ideas for addressing the challenges in bioacoustics, beyond providing a baseline model.

The ICLR 2024 call for papers explicitly mentions datasets and benchmarks in the list of relevant topics. As such, our contributions in designing an evaluation protocol which closely aligns with the real-world needs of ecological researchers and conservationists and demonstrating that it represents a significant and multi-faceted challenge stand on their own. In the evaluation framework we present, we encapsulate naturally occurring challenges, align these with important ML problem areas, and disentangle the influences of the underlying challenges into a comprehensive suite of tasks. Furthermore, in our initial empirical investigation and evaluation, we characterize the extent to which each provides challenges for the models we benchmark. Lastly, we provide inspiration to researchers by outlining concrete follow-up directions by enumerating our findings and open problems that stem from these.

The paper could be enhanced by a more extensive comparative analysis of different approaches or models for the tasks presented in the benchmark.

Within the extensive comparative analysis that we've presented in this work, are there any specific approaches or models that you think would be important to include in providing this first set of comprehensive evaluations, results, and analyses?

In regards to Weaknesses 3 and Question 1:

A meaningful insight from our work reveals the importance of using an appropriate learned representation in this framework when addressing bioacoustics retrieval. In particular, the model trained on XC upstream is more relevant and applicable to this task, providing a rich feature space that translates well to downstream tasks, much more so than the AudioSet dataset which includes many unrelated or irrelevant acoustics data. For domain adaptation, our conclusion is that we benefit from training on data that is more directly related than data that is less so. More explicitly, bird vocalizations provide a more useful representation for detecting other bird vocalizations than the broad and less relevant classes included in AudioSet.

[It] would be beneficial to consider additional evaluation metrics that are specific to bioacoustics and relevant to the benchmark's objectives.

ROC AUC is also common for bioacoustics tasks; refer for instance to Stowell et al. (2022): "Performance is measured using standard metrics such as accuracy, precision, recall, F-score, and/or area under the curve (AUC or AUROC)". Appendix A.3.2 provides a justification for our choice of metric. We will include a statement in the main paper to highlight this point.

Are there any specific experiments, ablations, or investigations that the authors plan to conduct in the future with the benchmark, beyond the preliminary baseline models presented in this paper?

As we touched on in response to your concerns in Weaknesses #1, our thorough investigation has provided meaningful open problems (in the context of foundational ML research and bioacoustics-related fields) and future work within this setting, as we have outlined in Section 6: Conclusion. Our group continues to explore problems within this space and are particularly interested in building self-supervised approaches that leverage large quantities of unlabeled and in-domain data.

Are there specific implications for domain adaptation, representation learning, or transfer learning from the challenges in bioacoustics and the proposed benchmark?

Implications include:

1. Using a relevant and general dataset for learning a rich and transferable feature space is a powerful approach. This is apparent from our comparison between XC models and AudioSet pre-trained models.
2. There is value in combining the baseline approaches we presented with domain adaptation techniques in building robust and generalizable retrieval models. This motivates the continued exploration of the intersection between domain adaptation, transfer learning, and bioacoustics, given that bioacoustics provides an inherently more complex backdrop than many academic domain adaptation settings.
3. As we previously stated, bioacoustics represents a rich playground to address fundamental ML questions but is underserved when focusing only on the canonical settings that are often explored, i.e. computer vision and language tasks.

We thank Reviewer SRto for their thoughtful feedback! We address the Reviewer’s questions, concerns, and feedback below.

If the label of a query is known, then what is the point of retrieving samples from a dataset?

The term "query" is used in the "retrieval" sense of the word: the query is not the example we wish to label, but rather the example (or handful of examples) which is a representative of a species’ vocalization that we seek to retrieve in the unlabeled corpus. This represents a practical use case where a practitioner might say: "I have an example of a Canadian goose vocalization (query) and want to find other examples of this species/vocalization type in my unlabeled dataset/field recording (search corpus).”

It would be nice to report results where none of the class embeddings from training time is used when evaluating the models.

Can you clarify what you mean by class embeddings from training time? If we understand correctly, here is some clarification. No examples (audio) used in evaluation are included at training time. This includes both the query examples and search corpora for each dataset/task. Furthermore, we exclude entire subsets of classes from being presented at training time.

I wonder if there is a universal class taxonomy which encapsulates all the classes in all the datasets.

In the empirical evaluation presented in this work, we produce a "standard" class taxonomy that encapsulates all the classes across every dataset we evaluate on, which is analogous to what you suggest and describe. It is worth noting that there is no universal taxonomy used across the field and that numerous class taxonomies exist and are updated every 6 or 12 months. Additional details on the topic of class taxonomies are included in the section titled "Labels & Species Codes" in Appendix A.3.1.

To measure "label shift", it is essential to know which classes have been seen during training vs at test time (i.e. seen and unseen classes).

We align with Moreno-Torres et al. (2012)'s definition of label shift (or prior probability shift): a problem for which P(Y) changes between the training and test datasets but P(X | Y) remains the same. In the experiment on disentangling label and covariate shift all classes are seen during training, but their marginal probabilities change in moving from the training set to the "evaluation (label-shifted)" set. In that setting there are no classes that are unseen at training time.

When measuring label shift, what is the semantic relation (due to their granularity) between seen and unseen bird classes?

Is your question in regards to a specific setting that we evaluate, as in Section 4.3 Generalization Results? Overall, the consideration of semantic relation is irrelevant/independent when measuring label shift. It is, however, relevant when considering performance on low resource classes (those with minimal representation at training), as in the case of the Artificially Rare task and matters when considering generalization to such classes. Additionally,

1. The ablation study includes all classes observed ("seen") at training time and investigates the impacts of label shift.
2. There is a component of label shift in the Artificially Rare task, and all bird classes within that collection are included at training time.
3. In the "Performance in mixed conditions" task, there is a combination of bird classes included at training time and those which are not present due to overlap with the Heldout Region sets.

Is it possible to evaluate domain-shift while fixing label-shift?

Absolutely! Refer to Figure 3: the "evaluation (label-shifted)" dataset is drawn from the same XC dataset as the training set, and therefore is not subject to domain shift, while the "Pennsylvania, USA" dataset is subject to domain shift (focal to passive). The "evaluation (label-shifted)" dataset's marginal class distribution is constructed specifically so as to match the marginal class distribution of the Pennsylvania dataset. Comparing the two allows us to assess the residual effect of domain shift while controlling for label shift. Note that it's not feasible to instead subsample the Pennsylvania dataset so that its marginal class distribution matches that of XC for two reasons: 1) some classes have a zero marginal probability in the Pennsylvania dataset, and 2) the Pennsylvania dataset is considerably smaller than XC, and evaluation noise would be unacceptably large if we were to subsample it.

Given that this is a benchmark paper, and that not everybody is super familiar with the domain (bioacoustics), I would expect a more direct and clear explanation of the task being solved, the types of input given to the network, etc.

Thank you for this feedback, we agree that we want to provide a helpful "table setting" for folks who aren't familiar with bioacoustics and provided overview in the Introduction and more in-depth explanation into this setting in the Appendix A.1 Why Avian Bioacoustics and A.2 Related Work. While we do describe in the main paper the base model training and inputs, along with the task description and breakdown thoroughly in the main paper, we can imagine that there would be value in having a high-level summary of all the components in either a figure, table, or brief highlight. Do you think that would help assure the reader has a clear sense of the elements of this framework? We would like to update the main paper to provide a bit more clarity for the reader.

Thanks for the response. However, my confusion still remains. For this reason I'd like to keep my initial score.

• As I mentioned in my review,

The evaluation benchmark retrieves relevant samples from a downstream dataset given a query, and it allows reusing class embeddings for queries learned during training. I'm not sure to get this point. I see no clarification regarding the use of class embeddings for "known/seen" classes. How can the identity of a sample (i.e., its class) already be known at test time? It can be predicted of course, but not mentioned in text, if I'm not mistaken.

• I strongly disagree with this argument:

Overall, the consideration of semantic relation is irrelevant/independent when measuring label shift

Semantic relation between classes/labels well impact transferability across them.

Thanks for updating the paper. It would have been nice to color all changes / new material in the updated version so that they can easily be tracked.