Towards A Translative Model of Sperm Whale Vocalization

Thank you for your review, which has helped us clarify and strengthen our paper. Most importantly, our work is novel as it is the first application of neural acoustic translation to cetacean vocalizations (only one other paper for neural bioacoustic translations exists, and it concerns birds). We also respectfully disagree with the assessment of limited technical novelty and lack of ablations; we will highlight where those are found in the paper, below. Next, we address each concern in detail and provide additional context about our technical contributions, evaluation methodology, and the significance of this work for both machine learning and marine biology communities.

Weaknesses

This paper focuses on developing a model for whale vocalization. However, the model has limited technical novelty in model architecture, training objectives. Their approach seems to consist of tokenization, token modeling, and generation. These modules are borrowed from previous work, VampNet, without modification, and therefore, this work has limited technical novelty. ... Lack of technical novelty and important observations is the main reason for my rating.

We respectfully disagree with this assessment and are glad to more specifically state the study's contributions. Firstly, there are very few examples of acoustic translation works on animal vocalizations (birds) - and this is the first to be applied to whale (or any cetacean) vocalizations. Our contributions include:

• Novel evaluation framework combining FAD, expert perceptual studies, and downstream tasks
• Largest curated sperm whale vocalization dataset
• Training strategies bridging music-pretrained models to underwater bioacoustics
• First demonstration that style transfer can preserve bioacoustic structure in cetaceans

It is not very clear if the success in the style transfer is quantitatively verified. In my understanding, FAD measures the distance between two distributions, yet does not measure the semantic similarity between the translated and original ones. Their generation results can be realistic enough to mimic the real codas, but they might not preserve the semantics of the input.

We appreciate this comment/concern and want to clarify a fundamental aspect of our work. As stated in the paper: "We emphasize that translation is in the acoustic sense; semantic translation remains a distinct and more ambitious goal."

For this reason, we use "acoustic translation" rather than "semantic translation." Understanding animal communication semantically would indeed be groundbreaking. For example, the 1973 Nobel Prize was awarded for decoding the semantic content of bee communication ("waggle dances"). No such understanding exists for whale communication.

Our experiments argue that our model preserves acoustic properties and temporal structure, not semantic meaning (which remains unknown to science).

The observations from their empirical results are limited. They combined several datasets to fine-tune VampNet and show that their trained model performs well on diverse whale-related downstream tasks. However, it is not clear what the key ingredient was, e.g., dataset combination or selection of VampNet. Table 1 only shows that their model outperforms the previous work. To give better observations and insight for readers, they need to provide more ablations into the combination of training datasets and base models.

Thanks for this comment, We want to point out that we have indeed provided extensive ablations in our paper. Specifically:

1. Section 4, final paragraph describes ablation experiments for domain adaptation and species-specific fine-tuning.
2. Appendix D repeats the FAD and downstream classification experiments while ablating on model components and dataset composition. Additional ablations have been carried out as a result of discussions with other reviewers, and will be added to the paper.

Anon1 and Anon2 seem to be their developed dataset. However, the explanation of the datasets is abbreviated from the main paper. Also, it seems it is not clear if they can publish the dataset. If they can, the novelty of this work should improve. ... They also do not mention that they will publish the dataset used to train their model. If their dataset is unique, they should focus on the presentation of the dataset.

We appreciate your interest in our dataset. If this paper is accepted, we will share the Anon1 dataset. We are in conversation with the data collectors of Anon2 regarding a partial or full release.

The paper focuses on the acoustic translation methodology rather than dataset presentation because: (1) the dataset combines two existing sources with our preprocessing pipeline, and (2) the methodological contributions (translation framework, evaluation suite) have broader applicability beyond this specific dataset.

We will also release our model weights and preprocessing code, enabling researchers to apply our methods to their own bioacoustic data.

We thank the reviewer for their enthusiastic assessment of our work and for recognizing the novelty of applying transformer-based architectures to sperm whale vocalization synthesis. Your technical questions have prompted us to conduct additional analyses that strengthen the paper and clarify important implementation details. Below, we address each of your concerns.

Regarding training a codec: Thank you for this insightful question about the fixed codec limitation. Your comment prompted us to conduct a detailed analysis of the codec's impact on sperm whale vocalizations. We performed systematic encoding/decoding experiments measuring frequency-dependent signal-to-noise ratio (SNR in dB):

• 0.2-3kHz: +0.15 dB
• 3-4kHz: -1.95 dB
• 4-5kHz: +0.15 dB
• 6-7kHz: -1.95 dB
• 7-9kHz: -2.77 dB

While we see some degradation in the 3-6kHz range (important for vowel-like features per Beguš et al.), the codec still preserves the primary coda energy reasonably well. The degradation is not severe enough to prevent our model from generating natural-sounding codas (Section 4.2), nor from its embeddings to "capture" vowels (Section 4.3).

Developing a specialized bioacoustic codec remains valuable future work, though computationally expensive. For now, these results suggest the fixed codec, while not perfect, is sufficient for demonstrating acoustic translation capabilities. We've added this quantitative codec analysis to our limitations section. Thank you for motivating this important addition.

Regarding echolocation clicks: You raise a valid concern. While our dataset curators made efforts to filter echolocation clicks from communication codas, some false-positives likely remain given the dataset's scale and technological limitations of click detectors. We will run a more recently-developed coda-detector on our dataset; if there are noticeable differences, we will retrain the model and release this as an additional checkpoint. Unfortunately, we will not be able to repeat the listener perceptual study.

It's worth noting that only one of five expert listeners reported this issue, suggesting it's not a pervasive problem in our generated outputs. The model appears to learn communication coda patterns preferentially despite possibly "contaminated" training data. We acknowledge this limitation in our dataset section.

Regarding fixing unnatural click properties: Thank you for this question. The artifacts mentioned by expert listeners have different solutions:

• Background noise: Standard bioacoustic preprocessing (DC offset removal, noise-gating) would remove this. Indeed, Beguš et al. (2024) apply such denoising to their sperm whale data.
• "Unnatural" clicks: This could be addressed through higher p-norm losses to emphasize acoustic events of varying magnitudes. However, this optimization likely needs to happen at the codec level, which is beyond our current computational resources. More training data may also help.

We deliberately chose to use raw outputs from WhAM without classical post-processing, as this paper focuses on demonstrating our neural model's generative capabilities (and limitations). For practical applications, standard bioacoustic denoising would be applied, but including it here would obscure the model's true performance.

We sincerely thank the reviewer for their thorough and constructive review, and for recognizing the potential impact of our work. We are particularly grateful for specific suggestions regarding bioacoustic embeddings and datasets, which have led to substantial improvements in our evaluation. Below we address each specific concern.

W4: Thank you for this excellent suggestion. We will retrain our model using BirdSet/iNatSounds for the domain adaptation stage. Given the larger data volume (100s-1000s of hours vs our current dataset), this will take considerable time but will be ready for the camera-ready version. We expect this enhanced domain adaptation to improve our results (and will report the results regardless of an improvement).

W5: This suggestion significantly strengthened our evaluation. We have now evaluated FAD using BirdNET [3] embeddings as you recommended. Results strengthen our results:

Analysis:

• WhAM dramatically reduces FAD scores across all non-target species (improvements of 50-110 points)
• Sperm whale FAD remains stable (-1.5 point difference is 1-2 orders of magnitude smaller than the rest, and could be due to disjoint test vs. reference sets)

We note that the difference in scale compared to Figure 3 is because we had to reimplement FAD ourselves in order to use these embeddings; the standard library we were using with CLAP was not compatible with BirdNET. Importantly, scale differences are irrelevant to FAD. These bioacoustic embeddings provide additional evidence for our model's effectiveness. Results will be added to the camera-ready version.

W6: Broadly speaking, the vocalizations of bearded seals (Erignathus barbatus), bowhead whales (Balaena mysticetus) and long-finned pilot whales (Globicephala melas) are "songlike" (trills, moans, groans, etc.). In that regard, acoustic impulses (even digital ones) are more similar to codas than these. If anything, this shows that FAD is not just picking up on background noise—which is relatively similar in these recordings as compared to the perfect silence of digital non-coda acoustic impulses. We will add this discussion to the paper.

We agree that FAD experiments alone would not be sufficient to support our overall claim regarding our model's acoustic translation capabilities. We chose to include them as they use an "objective" metric unlike the downstream classification and listener study experiments, which involve (to varying extents) human annotations.

W7: That is correct, accuracy refers to the success rate of the listeners. We will add arrows (\downarrow) to the y axis to emphasize that lower accuracy means better performance by WhAM.

W8: Your understanding is correct. The key difference is that in the 2AFC task, listeners directly compare a natural recording with the model's output on that exact same audio, allowing immediate A/B comparison. When the same acoustic content is presented in both versions, listeners can detect fine-grained artifacts that become apparent through direct comparison.

In contrast, the batch classification task requires identifying synthetic vs. natural audio without matched pairs. Listeners must rely on detecting absolute qualities that distinguish synthetic from natural speech across varied content, without the benefit of direct comparative cues. This fundamental difference in task difficulty explains the performance drop from ~80% to ~60%.

W9: Thank you for the great suggestion, we will add a sample of spectrograms to the camera-ready version. We will ask our study participants to annotate the spectrogram with their observations. We have already reached out and can confirm at least one participant is available to share their annotations.

W10: We agree that comparing against other embedding models would better demonstrate WhAM's utility. Following your suggestion, we evaluated BirdNET [3] and CLAP embeddings on our downstream tasks:

We emphasize that AVES, BirdNET, and CLAP are all encoder-only models designed for classification, whereas WhAM can be used for generation. As expected, embeddings trained on animal datasets (AVES, BirdNET) generally outperform general audio embeddings (CLAP) on more involved bioacoustic tasks.

Thank you for suggesting these additional experiments—they confirm the same trend across different embedding models, and add breadth to our analysis. The complete comparison table will be included in the camera-ready version.

W11: Apologies for the confusion. NCAIs = Training data (synthetic beeps/clicks that sound artificial) Synthetic codas = WhAM's output (intended to sound like real whale codas) We'll clarify this terminology throughout the paper.

W12: Interesting observation! Of the (near-)perfect scores in the spectrogram-assisted 2AFC, one was a marine biologist and another was an underwater acoustician. Interestingly, one is a professor with over a decade of experience, while the other is a PhD student. However, with n=5, we cannot draw statistical conclusions about expertise-performance correlation. We can surely suggest this as future work.

W13: In [ANONYMIZED LOCATION], Social Units are stable family groups of sperm whales - essentially extended matrilineal families that travel and socialize together. They represent the main ‘operational’ level of social structure of the whales lives, while mature males live predominantly alone, all females spend their lives in their natal units. SUs which share the same vocal dialect of codas make up a higher level of social structure called a vocal clan; however, as units move around the ocean there is a need to recognize one from another so there are some distinct features of their dialects that allow for this recognition [1]. This is likely what provides a cue for the classification task (see also [2]).

AVES's 92% accuracy in distinguishing between SUs could reflect: (1) genuine group-specific vocal characteristics from these stable family units, and/or (2) confounding factors from recording conditions, as different SUs were necessarily recorded on different days and locations. While (1) represents the biological signal we're interested in, (2) is a limitation inherent to field recordings in marine environments.

In conclusion, We believe these additional experiments and clarifications fully address your concerns. The new bioacoustic embedding results particularly strengthen our claims about both FAD metrics and downstream utility. Thank you again for your constructive feedback that has substantially improved our work.

—

Citations

[1] S. Gero, H. Whitehead, and L. Rendell (2016) Individual, unit, and vocal clan level identity cues in sperm whale codas. Royal Society Open Science 3: 150372.

[2] P.C. Bermant, M.M. Bronstein, R.J. Wood, S. Gero, and D.F. Gruber (2019) Deep Machine Learning Techniques for Sperm Whale Bioacoustics: Detection and Classification of Echolocation Clicks and Codas. Scientific Reports 9:12588

We thank the reviewer for their thoughtful and constructive feedback on our work. We appreciate your recognition of our interdisciplinary approach and the thorough evaluation methodology. Your questions have helped us identify areas where we can provide additional clarity and strengthen our empirical analysis. Below, we address each of your concerns in detail, including new experimental results that validate our design choices.

W (1): At time of writing the paper, we were unable to obtain samples from either of the GAN-based approaches: Kopets et al. did not have any public code or samples, while Begus et al. had no samples but did release public code; unfortunately, we were unable to run their code (we have reported the issues to the authors). If we cannot resolve the issue by the camera-ready, we will mention this in a footnote.

W (2): We appreciate this suggestion. We note that we already justified the necessity of domain adaptation through downstream classification experiments (mentioned at the end of Section 4 with details in Appendix D.2), which showed that neither domain adaptation nor SSFT alone produces useful embeddings. Following your suggestion for more extensive ablations, we have now repeated this analysis for FAD:

Importantly, when using the "SSFT-only" ablation on sperm whale inputs → sperm whale outputs, FAD increases to 0.63 (worse than average). This is a red flag—the model should preserve sperm whale acoustic properties when the input is already a sperm whale coda. In contrast, full WhAM achieves 0.29 FAD on the same task (which is the lowest FAD except for the reference dataset itself).

This suggests that without domain adaptation, the model cannot properly condition on input audio. Indeed, during development, we found that without domain adaptation, the model fails to generate coherent audio when translating from marine mammal inputs: outputs are so noisy that even non-expert listeners could distinguish them with ~100% accuracy.

Domain adaptation with diverse marine sounds (AudioSet, FSD) teaches the model to condition on input characteristics - a prerequisite for meaningful acoustic translation. Thank you for this suggestion - these FAD ablations revealed nuanced insights about the interaction between domain adaptation and species-specific training. We will add this interesting discussion to the ablation section of the paper alongside our existing downstream task ablations.

Regarding audio demos: We appreciate this suggestion and included audio samples as supplementary material with our submission. As a reminder, our supplementary materials include:

• Input/output pairs: Natural sperm whale codas and their WhAM translations for direct comparison
• Cross-species examples: Synthetic codas generated from both sperm whale and walrus inputs

To better serve readers unfamiliar with marine bioacoustics, we will enhance these materials for the camera-ready version. We will (1) add sample spectrograms directly in the main paper showing input/output pairs, (2) include a "listener's guide" explaining what acoustic features to focus on, and (3) add more diverse examples (different coda types, various acoustic conditions).

Additionally, we intend to release the full WhAM model on HuggingFace, enabling researchers to generate their own synthetic codas and explore the model's capabilities interactively.

Q (1): This is an excellent question that highlights a fundamental challenge in marine bioacoustics. We completely agree that conditioning on semantic or demographic information would be more scientifically valuable. However, the field faces severe data annotation limitations:

1. Only about 10% of all recordings made of the sperm whale units in our datasets between 2005 and 2024 have been manually annotated and verified and which contain behavioral annotations. We note that recordings vary in duration from ~4mins to hours long - and this is the largest curated sperm whale dataset known to us. More have had codas detected and annotated with new algorithms, but for this work, to be conservative, we focused on manually verified annotations only.
2. Speaker identity labels are rare as well: of the recordings which have been manually annotated only ~40% contain identified codas attributed to speakers which remains a very manual and time consuming step in the data annotation process.
3. The vast majority of recordings in our datasets are from females and immatures of both genders (who sound very much like their mothers dialect). In particular, they mostly do not contain mature male vocalization.

Our self-supervised acoustic translation approach was designed specifically to work with this reality—using audio context rather than requiring semantic labels. This establishes the technical foundation that future work can build upon as richer annotations become available.

Regarding clan conditioning: All our experiments use recordings from a single clan. Multi-clan synthesis would require years of additional fieldwork across ocean basins. We note that even aggregating our current single-clan dataset from two existing sources took several months due to format inconsistencies, data loss, and lack of standardization in the marine biology community.

Q (2): Excellent observation about the data-to-steps ratio. Allow us to clarify, and we will add this clarification to the model details in the paper. The key insight is that our training setup generates far more unique training examples than the raw hours suggest. At a high-level, this is how tokenization works:

• Each 2-second audio snippet becomes a 14×120 token array
• Columns represent time steps, rows represent acoustic granularity (coarse-to-fine)
• During training, we randomly mask entire columns (time steps)
• With 120 columns, there are 2^120 possible masking patterns per snippet

In our actual training data:

• 20 hours = 36,000 snippets × 2^120 possible maskings ≈ 10^40 possible training examples
• Training examples seen: 1M steps × 24 batch size = 24M
• Since 10^40 >> 24M, we effectively never see the same training example twice

During development, we subjectively evaluated model outputs every few thousand steps and observed a gradual shift from music-like sounds to authentic coda-like vocalizations. The model continued improving throughout the entire 1M steps without plateauing. If the reviewer believes quantifying this progression will significantly improve the paper, we can plot FAD scores versus training steps for the camera-ready version.