Representing speech through autoregressive prediction of cochlear tokens

We appreciate the reviewer’s acknowledgement of the originality, quality, and significance of the work. We thank the reviewer for their careful review. We provide clarifications to the reviewer’s comments/questions, and are currently implementing analyses suggested by the reviewer (training models on open-source data; verifying the efficiency of cochlear tokens), as clarified below.

Weakness 1. LFQ and Transformer lack biological basis.

We fully agree that not every part of our pipeline is supported by biological evidence, however, we also emphasize that the precise architecture underlying human auditory processing is not known, and hence we believe that LFQ and Transformers can serve as powerful stand-ins based on the heuristics detailed below.

LFQ: While LFQ itself might not be biologically inspired, we believe that our use of it in WavCoch has a biologically sensible interpretation: WavCoch implements a biological transformation (waveform to cochlear representation) and we contend that the LFQ bottleneck can be viewed as a readout of the discrete firing patterns of the neurons along this (largely unknown) pathway. The representations obtained from LFQ are binary 14-bit vectors which can be interpreted as an on-off neural firing pattern [1] which are then processed by downstream circuitry that is not yet well understood (also, please see a more detailed model diagram at the following link, bottom page, Figure 1: https://anonymous.4open.science/w/cochstream-project-page-0546/).

Transformers: We provide three reasons why we believe Transformers might serve as a useful approximation for stages of auditory processing:

i) It is well-established that prediction is a neural objective [2] and hence an autoregressive, causal Transformer is a reasonable candidate hypothesis for prediction-based processing in auditory processing at an algorithmic level, but likely not at the implementation-level [3].

ii) Transformers are incredibly powerful for many biologically-plausible tasks, and we wanted to build a model capable of handling various audio-based tasks that humans perform. In the perfect world, we would love to develop a model that perfectly aligns with neural circuitry at a fine-grained implementation-level and is powerful for tasks, but given that we do not know precisely the implementation-level details of auditory processing yet, we think the Transformer serves as a powerful stand-in.

iii) A large transformer with an autoregressive training objective can be viewed as evolving the circuitry that a biological brain is born with. Computational models did not undergo evolution in the same way as humans (or other species) did – hence, a large training phase where a generic model architecture is exposed to statistical regularities of the world (in this case, speech) may serve to wire up function-specific architecture similar to what a brain is born with.

Finally, we emphasize–as also mentioned in the discussion of our paper–that “We by no means claim that CochStream is a perfect biologically-inspired model, but it is a critical step in the right direction.”.

Weakness 2. Transformer-APC comparison.

We agree that a direct comparison between a Transformer-based APC model and CochStream would be useful, but given the large amount of additional experiments the reviewers have asked for and our limited resources we will unfortunately not be able to run all of them. Instead, we have included the following paragraph in the paper discussion to make it apparent that we do not claim a superior input representation but rather an advancement of previous work:

“WavCoch could operate with a mel spectrogram as the target instead of a cochleagram. Our work does not claim superiority of a cochleagram over a mel spectrogram. Instead, the main novelty from WavCoch lies in encoding one representation (in this case, the waveform) into another representation known to be computed in the auditory processing hierarchy (cochleagram) and “probing” an intermediary representation as discrete tokens. We chose the cochleagram for its biological plausibility. Hence, CochStream resembles the APC model [4], but with a quantized input and a Transformer architecture that enables efficient training and scalability.”

Nevertheless, as discussed below we will dedicate most of our resources to generating a version of our model trained purely on open-sourced data to improve the strength of several comparisons (see response to Weakness 7).

Thanks for the detailed clarifications and the effort to improve the experimental evaluations, also the updated diagram gives a much clearer picture of the architecture.

I still have some fundamental questions.

Regarding the "probing an intermediate representation as discrete tokens", isn't it the same as obtaining the discrete codes in any VQ-VAE or Neural Codec model, which converts the input representation to an output representation via a quantized bottleneck?

While it is common for VQ-VAE architectures to use the same input and output representations, there are established precedents where input and output representations differ. For example:

1. Spectral Codec [1]: Employs a spectrogram as input and a waveform as the output representation.
2. CLaM-TTS [2]: Employs mel spectrogram as output representation in its codec.

Given these examples, the work’s primary distinction appears to be the use of a cochleagram as the output representation. This biologically inspired choice is an interesting direction, but it primarily reflects an application-oriented novelty rather than a conceptual one.

That said, I am less concerned with the level of novelty and more interested in the justification for choosing cochleagrams over other well-established representations, such as mel spectrograms. While the benchmark results demonstrate that cochleagram tokens perform well, it remains unclear whether they offer specific advantages from a biological standpoint that are unique or particularly beneficial for the tasks addressed in the paper.

Key Questions for Clarification:

1. Comparative Performance: Would mel spectrogram tokens yield comparable results if used within the same architecture? If so, why prefer cochleagrams?
2. Unique Capabilities: Are there specific tasks, scenarios, or metrics where cochleagram tokens enable functionality that mel tokens cannot achieve? For instance:
   • Do cochleagrams provide better alignment with perceptual quality metrics?
   • Are they more robust to noise or distortions?

While I understand that many reviewers have already requested additional experiments, I want to emphasize that my intention is not to overburden the authors. However, providing some experimental or theoretical justification would significantly strengthen the impact of this work.

[1] Langman, Ryan, et al. "Spectral Codecs: Spectrogram-Based Audio Codecs for High Quality Speech Synthesis." arXiv preprint arXiv:2406.05298 (2024), https://arxiv.org/abs/2406.05298

[2] Kim, Jaehyeon, Keon Lee, Seungjun Chung, and Jaewoong Cho. "CLam-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech." The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=ofzeypWosV.

I really appreciate the author's efforts in performing the librilight experiments and the cochleagram inversion.

Time Critical Modifications before paper update deadline

1. The ablation of the vocabulary size is valuable, and shows the importance of a good ablation study, and may be more useful if included in the main text.
2. The table number and title should appear at the top of the table, I missed that in my previous readings of the paper and just realized it. (Mandatory)
3. Might I suggest including the appendix with the main paper?
4. Include the audio samples (video) as supplementary material if possible, because there is no guarantee that the anonymous link would persist, so maybe the audio samples and the appendix pdf (if not added to the main paper) could be zipped and uploaded as supplementary material.
5. I would be cautious about using the term novel framework because, in my opinion, the framework itself is not novel, a two-stage quantized representation followed by auto-regressive modeling is a fairly well-studied framework in several domains, the novelty lies in the specific output representation, and maybe the model architecture of the encoder-decoder. I would suggest the authors to rephrase.
6. Thanks for performing the experiments with mel-spectrograms, I think the results are reasonable. I would suggest putting them in a table instead of inline text, that will improve the readability.

Other Comments

1. I still did not get a satisfactory answer regarding some unique capabilities that might be enabled by using the biologically inspired representation.
2. Regarding ClaM-TTS, I agree that the mel-spectrogram is a pre-processing, and the model itself is an autoencoder. However, I disagree with the author's explanation of the Spectral Codec, from my reading of the paper, their input and out representations are different while the decoder architecture is based on hifi-gan, their encoder takes spectrogram and through the band-wise encoders and vector quantizers and the decoder converts them to the waveform, and trained with spectrogram based losses as well as adversarial loss. Thus, it also transforms one representation to another (spectrogram->waveform). Moreover, even in wavcoch, you extract the twiddle factors to first convert the waveform to time-frequency representation (which is not biologically inspired), so the wavcoch is more like transforming one TF representation to another TF representation. As far as I understood, the twiddle factor extraction is not trainable, hence can be considered a pre-processing step as well.

Overall, I think the paper is in a much better shape now, but there is still room for improvement.

I will increase my score to 5 for now. To increase it further, I would still need to be convinced about the potential unique benefits, response to point 2 in Other Comments and as much of the "Time Critical Modifications" to be done as possible in the limited time left.

Thank you so much for your continued engagement and valuable feedback.

We have implemented all of the time critical fixes to the paper itself as you suggested:

• We moved a short description of the WavCoch vocabulary size ablation to the main text and pointed to the Appendix for additional discussion and figure (for space constraint reasons).
• We fixed all the tables to conform to the format of caption above the body (thank you for catching this!)
• We merged the appendix into the main paper PDF and linked all the references.
• We created a zip file of all the sample videos and a README document and turned it into supplementary material. We plan on releasing a website for our project, that will have even more samples and will be hosted on GitHub (and thus be persistent), but it is nevertheless good to have a backup.
• We have modified the text to no longer refer to the two stage framework as novel, and have re-phrased all the places in which we claimed "novelty" of the framework. Instead, we explicitly refer to it as a "two-stage framework".
• We put the Mel spectrogram ablation results in a table.

After completing the above fixes in less than 24 hours after receiving your comment, we found that we are unable to upload another version of our PDF and supplementary material. We sincerely apologize for this. In order to provide a temporary remedy we uploaded a version of the paper with all the edits described above to our anonymous GitHub page here, as well as the supplementary zip file.

We once again truly apologize for this and promise to upload the fixed version for the camera ready submission upon acceptance.

We are very encouraged by the reviewer’s positive feedback and acknowledgement of the impact of the work, and are thankful for their thoughtful comments. Below, we clarify the reviewer's questions, and will provide an updated response when the suggested vocoder has been implemented.

Weakness. Reliance on ML benchmarks.

First, we appreciate the reviewer highlighting the importance of biologically-relevant benchmarking, which we agree demonstrates a significant gap in the literature. While ML benchmarks are of course standard for evaluating audio-based models, the lack of established biologically-relevant benchmarks becomes evident as (a subset of) the field moves toward more human-inspired models. Through this work, and in planned follow-up studies, we aim to contribute to closing this gap. We want to emphasize that we included the lexical similarity benchmark (sSIMI), which is not a fully standard ML benchmark as it involves direct comparison to human performance. Hence, this represents an initial step toward incorporating more human-based evaluations.

Second, as the reviewer mentions, we believe that a fundamental advantage of CochStream is that it can naturally be prompted with sounds (much like a language model can be prompted with text). This capability can indeed pave the way for a new era of cognitively inspired prompt-benchmarking of auditory models. Since our model is unique in this capability we were unable to compare it to any baselines, however we will attempt to perform a pilot evaluation of the continuations by training a vocoder to obtain audible continuations (also see response to Question 2 below) during the rebuttal period. We will include a detailed discussion of this paradigm and its implications in the final manuscript.

Third, if the reviewer has a particular biologically-inspired evaluation in mind for the current paper, we would be happy to test our model on it (depending on the benchmark’s scope, of course). Otherwise, we are very interested in pursuing this direction in future papers, once we have introduced the model in the current paper.

Question 1. Performance on non-content tasks.

We will provide a more thorough analysis of our model’s performance characteristics and unique advantages by the end of the rebuttal period, including addressing the disparity between the performance on different types of tasks.

Question 2. Speech continuation capability.

To perform some preliminary analysis of our model’s speech continuation capability, we are currently training a vocoder model to decode the continuations back to audio. We will show some preliminary results by the end of the rebuttal period and will include a deeper analysis, including qualitative human evaluations, in the final manuscript.

We also want to emphasize that the focus of the current paper is speech representation learning, and not language modeling and hence we leave the longer continuations and linguistic benchmarks for a separate paper.

Question 3. Motivation for autoregressive training.

We chose the autoregressive training objective for three main reasons (which we will include in the discussion of the revised manuscript):

i) Amongst the algorithms commonly used in modern AI, we believe autoregressive learning is the most biologically plausible one. Unlike the BERT objective, it does not violate causality, and does not require us to discover units through global k-means clustering for subsequent use in autoregressive sequence models. Unlike contrastive objectives it does not require a large selection of heuristically-picked positive and negative examples. Instead it simply predicts the distribution of next plausible sounds.

ii) Autoregressive training allows for the evaluation of the surprisal associated with any given sequence of sounds, and for the speech continuation capability discussed in Question 2 above.

iii) Autoregressive training is simple, yet capable of capturing enormous complexity when appropriately scaled. In the paper, we demonstrate that the autoregressive objective scales well within the speech domain (comparison of CochStream-base versus CochStream-large).

Thanks again for the time spent reviewing our work. We have now addressed all of your questions and concerns through the individual response (posted last week) as well as through the global response. If you have any further questions that are preventing you from raising our score, please let us know and we will do our best to address them in the extended discussion period.

We would once again like to thank the reviewer for their comments and suggestions. We have addressed each one of the concerns raised by the reviewer either through a text response or with follow up experiments.

Specifically, we:

• Developed a procedure for inverting the predicted cochleagrams into audio, and generated several rollout samples available at the bottom of out anonymous website (https://anonymous.4open.science/w/cochstream-project-page-0546/). We really hope the reviewer will take a look at these, as they provide great intuition about the model’s predictions and the interpretability advantage of our framework.
• Responded with clarifications behind our reasoning for choosing the autoregressive training objective. We also wanted to offer another clarification on task-performance: Some of the SUPERB task performance patterns, where the cochleagram representation did not yield the expected performance characteristics, can likely be attributed to the naturalistic dataset used to train our initial model. To address this, we trained a new version of the model on the Libri-Light dataset, which is also used by the baseline comparison models. Although we could only evaluate the halfway checkpoint due to time constraints, this version demonstrated strong performance compared to the model trained on our original dataset. We anticipate that the fully trained model will outperform the initial version on many of the SUPERB tasks. As a final point, we note that the Libri-Light dataset, comprising clean recordings of English audiobooks with single-speaker consistency throughout, offers characteristics particularly well-suited for pre-training certain tasks. Previous research has shown that the choice of training data can be more critical than model architecture or task objectives in shaping model representations [1]. In the camera-ready version of this paper, we plan to present a comprehensive ablation study examining how different datasets influence task performance. Additionally, we will explore a “best of both worlds” approach by training a model on a combination of our naturalistic data and the Libri-Light dataset.

Since it is getting close to the end of the rebuttal period and we believe that we have addressed all of the stated weaknesses we kindly ask the reviewer to let us know if there are any further unaddressed concerns–otherwise we would be grateful if you could adjust the score accordingly.

References: [1] Conwell, C., Prince, J.S., Kay, K.N. et al. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nat Commun 15, 9383 (2024). https://doi.org/10.1038/s41467-024-53147-y

Weakness 5. Autoregressive modeling does not capture the complexity of biological systems.

We agree with the reviewer’s point: it is likely that an algorithm that captures the full complexity of biological systems might need to include different kinds of objectives, recurrence, wiring constraints, different cell types etc. – that said, very little is known about the neural architecture and learning rules underlying auditory processing in humans, and hence we believe that we cannot a priori reject autoregressive learning. Below, we emphasize three points as to why autoregressive Transformers might serve as a useful approximation in the current context:

i) It is well-established that prediction is a neural objective [1] and hence an autoregressive Transformer is a reasonable candidate hypothesis for prediction-based processing in auditory processing at an algorithmic level, but likely not at the implementation-level [2].

ii) Transformers are incredibly powerful for many biologically-plausible tasks, and we wanted to build a model capable of handling various audio-based tasks that humans perform. In the perfect world, we would love to develop a model that perfectly aligns with neural circuitry at a fine-grained implementation-level and is powerful for tasks, but given that we do not know precisely the implementation-level details of auditory processing yet, we think the Transformer serves as a powerful stand-in.

iii) A large transformer with an autoregressive training objective can be viewed as evolving the circuitry that a biological brain is born with. Computational models did not undergo evolution in the same way as humans (or other species) did – hence, a large training phase where a generic model architecture is exposed to statistical regularities of the world (in this case, speech) may serve to wire up function-specific architecture similar to what a brain is born with.

Finally, as we emphasize in the discussion of the paper, “We by no means claim that CochStream is a perfect biologically-inspired model, but it is a critical step in the right direction.”. We believe our paper is a big step in the right direction through i) more biologically plausible input representations compared to other quantization schemes, ii) no global clustering procedures, iii) no large intra-batch comparisons, and iv) no bidirectionality. In the discussion, we additionally refer to several works that argue that Transformer computations can be implemented in biological tissue [3-5].

References
[1] Spratling, Michael W. "A review of predictive coding algorithms." Brain and cognition 112 (2017): 92-97.
[2] Marr, David, and Tomaso Poggio. "From understanding computation to understanding neural circuitry." (1976).
[3] Bricken, T., & Pehlevan, C. (2021). Attention approximates sparse distributed memory. Advances in Neural Information Processing Systems, 34, 15301-15315.
[4] Whittington, James CR, Joseph Warren, and Timothy EJ Behrens. "Relating transformers to models and neural representations of the hippocampal formation." arXiv preprint arXiv:2112.04035 (2021).
[5] Kozachkov, Leo, Ksenia V. Kastanenka, and Dmitry Krotov. "Building transformers from neurons and astrocytes." Proceedings of the National Academy of Sciences 120.34 (2023): e2219150120.

We thank the reviewer for their positive feedback and insightful questions! Below we describe experiments investigating vocabulary utilization rate and answer questions requiring conceptual clarification.

Weakness 1. Baseline models seem weak.

We will include an audio autoencoding (neural codec) model baseline as the reviewer suggested. We believe that Whisper is not a fair comparison given that it is trained on labeled text data. Instead, we believe that other self-supervised models are fair comparisons.

Weakness 2. Main advantages of CochStream.

We are working on including additional comparison models per the reviewers’ comments, and following these results, we will provide a more thorough analysis of our model’s advantages by the end of the rebuttal period.

Weakness 3. Vocabulary utilization rate of WavCoch.

We thank the reviewer for raising this point. We would like to clarify that the cochlear tokens are meant to be a lightweight encoding of the underlying auditory stream (much like BPE tokens for text), not a deep semantic clustering of features. Nevertheless, we conducted some cluster purity experiments and surprisingly found that even at the token level the model starts to extract structures that are loosely related to phonemes.

We compiled some initial results into a plot which can be found at the bottom of our anonymous website (https://anonymous.4open.science/w/cochstream-project-page-0546/) as Figure 2. First of all, we asked how many tokens were utilized for the out of distribution TIMIT test set. We found that 82.5% tokens were utilized (13,517 tokens out of 16,384).

Second, we investigated the vocabulary utilization rate in relation to the phoneme purity of each token. For each token, we define the purity as:

Purity = (Count of most associated phoneme for token i) / (total counts for token i)

Hence, the purity (on Figure 2’s y-axis) shows the purity for each token, whereas the color shows how many times a given token appears in the test set. As evident from the plot, even tokens that occur more than 70 times each have high purity values, up to 0.85.

Inspired by these initial results we will conduct a more thorough investigation of the cochlear token space, with varying bottleneck sizes during the course of the rebuttal period. We will include a detailed section on these results in the revised manuscript.

Weakness 4. Long continuations based on a seed.

It is correctly observed that CochStream generates relatively long waveform continuations that appear realistic. To perform some preliminary analysis of their quality, we are training a vocoder model to decode the continuations back to audio. We will show some preliminary results by the end of the rebuttal period and will include a deeper analysis in the final manuscript.

We also want to emphasize that the focus of the current paper is speech representation learning, and not language modeling and hence we leave the longer continuations and proper linguistic evaluations for a separate paper.

Thanks again for the time spent reviewing our work. We have now addressed all of your questions and concerns through the individual response (posted last week) as well as through the global response. If you have any further questions that are preventing you from raising our score, please let us know and we will do our best to address them in the extended discussion period.

We would once again like to thank the reviewer for their comments and suggestions. We have addressed each one of the concerns raised by the reviewer either through a text response or with follow up experiments.

Specifically, we:

• Added additional baselines to our revised results tables (see Tables 1 and 2 in the revised paper).
• Added additional information regarding the advantages of the cochlear representation and our model more broadly, as well as comparisons to more recent approaches in the field (see Section 4).
• Ran ablation experiments to discover the optimal vocabulary size for our model (see Appendix Section I.3) This actually resulted in us finding an improved vocabulary size for our tokenization model (one bit smaller than the previous one) – thank you!
• Developed a procedure for inverting the predicted cochleagrams into audio, and generated several rollout samples available at the bottom of out anonymous website (https://anonymous.4open.science/w/cochstream-project-page-0546/). We really hope the reviewer will take a look at these, as they provide great intuition about the model’s predictions and the interpretability advantage of our framework.
• Wrote a lengthy reply to questions regarding to what extent the simple autoregressive modeling approach can capture the complexity of biological systems in our initial reply (https://openreview.net/forum?id=TQdg1X6eqm&noteId=MOlIO5Qbhe) and made modifications to the discussion section of the paper (see Section 4).

Since it is getting close to the end of the rebuttal period and we believe that we have addressed all of the stated weaknesses we kindly ask the reviewer to let us know if there are any further unaddressed concerns–otherwise we would be grateful if you could adjust the score accordingly.

Questions 1 and 2. Issue: Lacking information about CochStream embeddings.

We apologize for the oversight. Here is a description, which we have of course included in the revised manuscript:

We obtain CochStream embeddings by pooling the embeddings of all the tokens associated with the corresponding temporal section of the cochleagram via ground-truth phoneme or word boundaries (Section 3.1). For the pooling operation, we tested mean/max/min pooling for the linear probing experiments and lexical similarity for CochStream and the comparison models.

Finally, to answer the reviewer’s question related to how obtaining embeddings from GPT-style models like CochStream relates to masked models like HuBERT:

When analyzing a sound (such as when we obtain embeddings for a given phoneme/word), all computations are performed in parallel (using a causal mask, like during training) eliminating any inference time bottlenecks caused by the causality constraint. We note that our model is of course also capable of generating long-form continuations of audio, which does take longer to rollout (much like an LLM) – however this is a capability that is not present in competitor models such as HuBERT leaving no clear point of comparison.

References
[1] Cooley, James W., and John W. Tukey. "An algorithm for the machine calculation of complex Fourier series." Mathematics of computation 19.90 (1965): 297-301.
[2] Yu, Lijun, et al. "Language Model Beats Diffusion--Tokenizer is Key to Visual Generation." arXiv preprint arXiv:2310.05737 (2023).
[3] Feather, Jenelle, et al. "Model metamers reveal divergent invariances between biological and artificial neural networks." Nature Neuroscience 26.11 (2023): 2017-2034.
[4] Radford, Alec. "Improving language understanding by generative pre-training." (2018).

We thank the reviewer for their careful review and thoughtful feedback! Below, we address all the reviewer’s questions (in two separate comments), except for the comparison to additional baseline models and a more thorough analysis of model performance, which we are currently working on. We are happy to further clarify any of the answers below.

Weakness 1. Lacking clear explanations of the cochlear encoding process, its relation to mel spectrograms, and model diagrams.

We have created a detailed model diagram, demonstrating the WavCoch architecture as well as the CochStream architecture, please see Figure 1 uploaded at this anonymous link (bottom of the page): https://anonymous.4open.science/w/cochstream-project-page-0546/.

We will include the model diagram in the revised manuscript along with a detailed description:

WavCoch Architecture: First, the raw waveform (shape: 1,80000 for 5s of mono audio sampled at 16kHz) undergoes the Fourier Transform by computing Twiddle Factors [1]. These factors represent complex sinusoidal components that decompose the signal into its frequency spectrum. The Twiddle Factors are applied to the audio signal through a 1D convolution (window size 1,001 and hop length 80 samples) which transforms the signal into the time-frequency domain.
Second, each 5 ms temporal step of this time-frequency representation is fed into two fully-connected (FC) layers with ReLU nonlinearities (with 512 hidden units each).
Third, these embeddings are then passed through a 14-dimensional LFQ bottleneck [2], which effectively binarizes the representation. We read out the activations of this bottleneck as a 14-bit binary code which can be interpreted as one of 2^14 = 16,384 discrete tokens.
Fourth, the output of the LFQ bottleneck is then projected to a 211 dimensional output, through two 1-dimensional convolutional layers (kernel size 10 and stride 1), separated by ReLU nonlinearities. This output corresponds to the frequencies in the cochleagram representation [3] which it is supervised to match via L2 error. Thus for every 5 seconds of audio, WavCoch extracts a sequence of 988 integers in the range [0, 16384) through the LFQ bottleneck, denoted as cochlear tokens, to feed into CochStream.

CochStream Autoregressive Model Architecture: The cochlear tokens obtained in WavCoch are passed to a GPT-style autoregressive Transformer [4], denoted as CochStream. We train two versions: CochStream-base (97M parameters), with 12 layers, 12 attention heads and an embedding size of 784 and CochStream-large (1.3B parameters) with 24 layers, 16 attention heads, and an embedding size of 2,048. Both models have a vocabulary size of 16,384. The CochStream model takes as input the cochlear token sequence produced by WavCoch and predicts the next token in the sequence. The context length is approximately 20s (4,096 tokens). We utilize a learned positional embedding and compute the cross-entropy loss between the predicted logits and the true next token in the sequence.

We are very happy to supply the actual implementation to the reviewers if they wish, and will publicly release the full code base upon acceptance.

Finally, we clarify WavCoch’s relation to a mel spectrogram: WavCoch could operate with a “simple” mel spectrogram as the target instead of a cochleagram. We do not make any claims about the superiority of a cochleagram over a mel spectrogram (we have clarified this in the revised manuscript). Instead, the main novelty from WavCoch lies in encoding one representation (in this case, the waveform) into another representation known to be computed in the auditory processing hierarchy (cochleagram) and “probing” an intermediary representation as discrete tokens. We chose the cochleagram for its biological plausibility.

The cochleagram implementation that we use can be found online in the following public repository, which we will link to in our revised manuscript along with a brief description: https://github.com/jenellefeather/chcochleagram.

Weakness 2. Issue: Linear probing performance lacks relevant models.

We agree that smaller models would be helpful. We are currently working on including the models the reviewer suggested (WavLM-base or wav2vec2-base) for comparison the in linear probing results. Following these results, we will also provide a more thorough analysis of our model’s performance by the end of the rebuttal period.

We would once again like to thank the reviewer for their comments and suggestions. We have addressed each one of the concerns raised by the reviewer either through a text response or with follow-up experiments.

Specifically, we:

• Added a diagram depicting architectural details of our pipeline, as well as a detailed description (See Appendix I.1).
• Added additional base models to compare against CcohStream-base (see Tables 1 and 2). We further discussed advantages and disadvantages of our model (see Section 4).
• Added information to the “2.1.4 OBTAINING COCHSTREAM EMBEDDINGS” section from our manuscript as it was erroneously left blank in the initial submission.

Since it is getting close to the end of the rebuttal period and we believe that we have addressed all of the stated weaknesses we kindly ask the reviewer to let us know if there are any further unaddressed concerns–otherwise we would be grateful if you could adjust the score accordingly.

Mel spectrogram vs. cochleagram

As raised by reviewers ipew and Nhso, we trained a version of our WavCoch model using a standard 80 mel-spectrogram representation (instead of a cochleagram representation). We trained it on the publicly available librispeech960 dataset, consisting of 960 hours of speech recordings [1]. Since the spectrogram L2 reconstruction error is not directly comparable between a cochleagram and mel-spectrogram we utilize two proxy measures: i) Number of unique codes utilized and ii) Phoneme cluster purity. Both of these metrics are computed on the out-of-distribution TIMIT test set. First, related to the number of unique codes utilized: We find that the WavCoch model trained with mel spectrograms utilized 8151 out of 8192 codes, while the cochleagram version utilized 8172 codes. Second, related to phoneme cluster purity: We find that the mel spectrogram model achieved an average phoneme cluster purity of 0.3473 while the model trained with the cochleagram achieved an average phoneme cluster purity of 0.3517. While these results are preliminary, they suggest that the cochleagram representation performs at least as well as, if not slightly better than, the mel spectrogram in this context. We are continuously exploring this comparison further, and will report the findings in the final version of the manuscript. Finally, besides the quantitative analyses reported above, we prefer the cochleagram over the mel-spectrogram representation for conceptual reasons: The ultimate goal of our framework is to move towards more biologically plausible speech models, and the cochleagram is more aligned with this goal.

In the revised paper, we have added a discussion on the mel-spectrogram versus cochleagram issue and explicitly note that we do not claim the superiority of cochleagrams over mel-spectrograms.

CochStream Libri-Light task performance analysis

As raised by reviewer Nhso, we trained a version of CochStream using the 60k hour Libri-Light dataset [2] (denoted as CochStream-large-ll in the updated version of the paper). We utilized cochlear tokens produced by the new WavCoch model with a vocabulary size of 8192 which proved superior in the experiments mentioned above. We also adjusted the CochStream Transformer parameters to exactly match those of HuBERT (and WavLM) for an even more apples-to-apples comparison (suggested by e.g., reviewer ipew). While our full training schedule is set for 500k optimization steps, we were not able to complete it in time for rebuttal due to time and computational constraints. Nevertheless, even at 200k steps our model already matches the performance of the model we described in the initial submission. We highlight some of the results below:

On the TIMIT linear probing task the libri-light model matches the performance of the one trained on our data on phoneme decoding (92% old vs. 92% new) while it outperforms it on the word decoding task (67% old vs. 69% new). On the ZeroSpeech 2021 Lexical Semantic Benchmark the libri-light model scored above the old base model and all other baselines we evaluated, but below the large model trained on our dataset. This suggests that for this task, both our modeling approach and our dataset are important for maximizing the performance. On the SUPERB benchmark the libri-light model faired comparably to the model trained on our data.

For the full results please see tables 1, 2 and 3 in the updated main paper pd.

We believe that these results demonstrate that the core performance contribution of our work comes from our novel two-stage modeling framework (WavCoch tokenization followed by CochStream auto-regressive prediction). Furthermore, this demonstrates that our model is not contingent on one single dataset – we obtain very competitive performance on different audio datasets. We are continuously evaluating our models on SUPERB, and will send a further update in the extended discussion period about performance characteristics as suggested by reviewers.

Improvements To Address Concerns:

Open-source data reproduction

We thank reviewer Nhso for their suggestion, and in line with our strong commitment to open science, we have reproduced key results using a model trained on fully openly available datasets. First, we retrained a version of WavCoch using the librispeech960 dataset consisting of 960 hours of read English speech [1]. Second, we train a 1B parameter version of the sequence-to-sequence CochStream model on the full 60k hours of the libri-light dataset consisting of 60,000 hours of unlabeled English speech [2]. As demonstrated here, we show that this model demonstrates strong performance on a similar set of tasks than the version trained on our dataset (see Section below titled “CochStream Libri-Light task performance analysis”). This model serves as a strong replication of the results from our initial model using popular open-source datasets.

WavCoch vocabulary size analysis

As suggested by reviewers Nhso and npk7, we performed ablations on the vocabulary size of the WavCoch model. In an attempt to ensure maximal reproducibility, we perform these ablations using the librispeech960 dataset as mentioned above. We train variants of WavCoch using a vocabulary size of 16384, 8192 and 4096 (14, 13 and 12 bit codes respectively). We report the results in Figure 3 (https://anonymous.4open.science/w/cochstream-project-page-0546/) which shows the phoneme cluster purity and reconstruction error on an out of distribution test set (TIMIT test set). We introduced the cluster purity metric in our original response (https://openreview.net/forum?id=TQdg1X6eqm&noteId=7OxgsydgYp) which is defined as purity = (Count of most associated phoneme for token i) / (total counts for token i) which intuitively provides a metric for how consistently a given token aligns with a specific phoneme.

The plot shows that a vocabulary size of 8192 presents both the highest cluster purity and lowest reconstruction error. Furthermore, this vocabulary size presents a local minima in the MSE space, and a maxima in the cluster purity space indicating it is the optimal size for generalization in this domain. We are very grateful to the reviewers’ suggestion as it revealed an improvement in our model design. We will use a vocabulary of 8192 for all future models.

Vocabulary token clustering analysis

As suggested by reviewers Nhso and npk7, we analyzed the distribution of tokens associated with specific phonemes in the TIMIT test set. We visualized the distribution in Figure 4 ( https://anonymous.4open.science/w/cochstream-project-page-0546/) and found that multiple tokens are typically associated with each phoneme. Notably, a large number of tokens are linked to the "sil" (silence) label, but we would like to clarify that this label includes any non-speech sound, not true silence. Since WavCoch is optimized with a single objective–to predict the cochleagram as best as possible–it has no direct incentive to collapse the codebook by merging codes associated with the same phoneme. However, our analyses show that this clustering takes place naturally, to an extent. In future work, we plan to investigate WavCoch models that explicitly incorporate a more efficient codebook, and compare it against the current baseline.

Interpretability of speech continuation capability

As suggested by reviewers npk7 and 6c7S, we investigate the decoding of CochStream predictions back into the auditory signal. To this end, we developed a simple procedure for inverting the cochleagrams back into a waveform. This procedure is a per-sample optimization of making the waveform match the cochleagram prediction.

Specifically, we optimize a 1D tensor representing the waveform input to make its cochleagram representation match the cochleagram predicted by WavCoch (via L2 error). We backpropagate through the cochleagram transformation and use the Adam optimizer with a learning rate of 1e-2. Note that this optimization procedure is not a learned vocoder model, but a simple procedure of converting the output of WavCoch, the cochleagrams, into audible sound (conceptually similar to Griffin-Lim algorithm [3]).

We upload several audible samples of speech generations from our CochStream: (https://anonymous.4open.science/w/cochstream-project-page-0546/). Please access the page using Google Chrome or Firefox as we have seen some cases of Safari not properly loading these videos. We truly hope that the reviewers will take a few seconds to listen to the continuations.

We observe that on short time-scales the model produces reasonable completions, but the longer the completion, the more the predictions drift away from being plausible. We would like to emphasize that the purpose of CochStream is not to be a language model, but a speech representation model – the fact that it can perform rudimentary language modeling is a serendipitous side effect of the training objective, which points to the fact that understanding speech, and producing language can be thought of as a unified objective. These findings serve as great motivating factors for a planned follow-up work which will attempt to stabilize longer term speech generations, building on top of the foundation laid out in this paper.

References:

[1] Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015, April). Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 5206-5210). IEEE.

[2] Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P. E., ... & Dupoux, E. (2020, May). Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7669-7673). IEEE.

[3] Griffin D. and Lim J. (1984). "Signal Estimation from Modified Short-Time Fourier Transform". IEEE Transactions on Acoustics, Speech and Signal Processing. 32 (2): 236–243. doi:10.1109/TASSP.1984.1164317