Disentangling Textual and Acoustic Features of Neural Speech Representations

Many thanks for your detailed review!

[Experiments comparing the proposed method with existing approaches are lacking. As there are lots of works for speech representation disentanglement like AutoVC, SpeechSplit, or FAcodec, it would strengthen the paper to report the performance of at least one existing method]

Thank you for raising this point – we regret not having been more explicit about this in the original submission. The key point to make here is that, while it is true that there is lots of work for speech representation disentanglement, our work is not directly comparable to them. In our study, we specifically posit the following desiderata:

• The approach must be able to disentangle representations already learned by large pre-trained models of spoken language, with minimal extra data and limited computational resources. We thus do not consider methods that rely on learning disentangled representations from scratch, or require large amounts of fine-tuning data.
  • For example, AutoVC, SpeechSplit, Prosody2Vec, and FACodec, all have their own complex modeling architecture (including a deep stack of CNNs and LSTMs), and trains on raw audio to reconstruct the input audio signals, and this is why they are evaluated for style transfer in voice conversion application.
  • For example, as stated by the authors in the paper, the entire pretraining procedure for Prosody2Vec takes around three weeks, as their parameters are first pre-trained on spontaneous and emotional speech and then fine-tuned for the target task (emotion). In contrast, we train our proposed method directly on the target task, and it takes only a few minutes.
• The approach must be general and applicable to representations learned by pre-trained models with different architectures, sizes, and learning objectives. We thus do not consider methods that are tailored to a specific architecture.
  • e.g., Prosody2Vec which builds on top of HuBERT's quantization module to encode content.
• The approach should allow for acoustic features to emerge based on their relevance to a target task (such as emotion or speaker identification), rather than being shaped around pre-specified static features such as pitch and timbre. This last point is important because defining acoustic dimensions in advance might result in losing some task-relevant aspects of acoustic representations.
  • For example, existing work pre-specify the following components regardless of the target task:
    • AutoVC: content, style
    • SpeechSplit: content, timbre, pitch, rhythm
    • Prosody2Vec: content, speaker, prosody
    • FACodec: content, prosody, timbre, and residual acoustic

The above desired properties put strong constraints on our approach and distinguish our study from all the existing work on learning disentangled representations for speech that we are aware of. We do, however, present a serious effort to compare to FACodec and SpeechSplit in our revised manuscript (Appendix F);

Thank you for your response.

The rebuttal addresses most of my concerns and highlights an additional strength I previously overlooked: the efficiency of training in terms of dataset size and time consumption. The revisions strengthen the paper.

However, I still have a major concern:

Q2: [The WER reported in Table 1. seems to be higher than expected. What is the possible reason for that?]

Pre-trained models do not have very strong transcriptional information (the final layers do not have any sense of transcription, and most of the information comes from the middle layers). During the common ASR fine-tuning process, these last layers undergo significant changes. Thus, learning a latent representation that is robust to noise and discards irrelevant information makes pre-trained representations more resilient and less sensitive to noise, allowing them to perform better than entangled representations.

Specifically, my concern relates to the WER of VIB when applied to HuBERT-FT and Wav2Vec2-FT. I think they should achieve much lower WERs on LibriSpeech and Common Voice datasets (as observed in Probing tasks).

For the reasons mentioned above, I will raise my score to marginally above the acceptance threshold, as I believe the motivation and the efficiency of the proposed method are strong. However, it appears to somewhat compromise textual content information.

Many thanks for your detailed review!

[The motivations in the abstract and conclusion are not well connected to the modeling and analysis. E.g. one motivating application is to minimize the privacy risk from encoder representations. This hasn't been assessed in the model or paper]

Thank you for this feedback. We clarified the motivation in the introduction and removed the reference to the privacy risks.

[It is unclear how these learned representations would transfer to some new task. Would this approach need to be extended to a "stage 3 training process?]

There is no need for a stage 3. The textual latent representations produced in stage 1 are independent of the target task and can be applied to new downstream tasks. As a result, for each new task, one can directly start with stage 2 to train the shallow encoder and classifier on the target task. We believe this is, in fact, a strong point of our approach, as it reduces the required compute considerably when there are many such new tasks.

[Multiple training stages incur additional complexity. It would be interesting to see if these multiple objectives would be included into a single stage training]

Joint training is certainly feasible. However, training in stages offers key advantages: Stage 1 can be trained once and reused across all tasks, while Stage 2 training is highly computationally efficient. Moreover, we are publicly releasing the Stage 1 models, enabling follow-up work to easily build on our method by focusing solely on Stage 2 training. From a paper perspective, this approach simplifies the discussion of results, ensuring comparability and robustness across tasks since they all share a common starting point in Stage 1.

[Inconsistent task performance improvement/degradation for different objectives and model sizes]

First and foremost, we emphasize that our paper primarily focuses on disentanglement. These experiments were conducted to demonstrate that the disentanglement method does not result in any significant trade-off in performance in downstream task performance (Emotion and Speaker Id). For transcription (which is not the focus of this work), increase in WER from using VIB is expected for models fine-tuned on large datasets. This is because stage 1 was conducted on smaller datasets, which can inadvertently suppress some of the WER capabilities. Apart from this, identifying a clear pattern may be challenging. We could speculate that there is a trade-off at play: disentangled representations can facilitate the extraction of task-specific information, thereby enhancing accuracy, but the information bottleneck can also to the loss of potentially beneficial information, which could negatively affect accuracy. This trade-off likely varies depending on the model's architecture and size. While the difference in performance varies, the results collectively suggest that probing has a largely neutral impact on model performance while offering the added benefits of interpretability and control.

Q1: [Why subsample Librispeech and Common Voice so heavily for the transcription task? Librispeech contains 960h of transcribed audio, but this approach uses less than 20]

In our initial experiments, we found that a few hours of transcribed data were actually sufficient to train a classifier for decoding transcription. According to Tables 9 and 10 of Appendix C in Wav2Vec2 1, using even only 1 hour of transcribed data is enough to achieve ASR performance comparable to fine-tuning on the full 960 hours. Please note that in contrast to ASR fine-tuning, the model's weights in our approach are fixed; we only train an encoder bottleneck together with a linear classifier. While increasing the scale of transcribed data in stage 1 might improve CTC performance, it would not harm the core observation here, as we compare the performance using the entangled representations (probing) with the disentangled ones (VIB), both of which utilize the same transcribed data.

Q2: [How important is the ordering of the tasks? Would the performance be identical if Emotion or Speaker Id were stage 1 and Transcription was stage 2?]

Indeed, it matters! For example, when it comes to emotion, both the content of the audio (text) and the acoustic features can help recognize emotion. Our goal here is to measure the contribution of each of the textual and acoustic features to the target task. To address this, we first separate the textual features at stage 1, and then focus on finding the remaining features at stage 2, which go beyond text but actually contribute to the target task. Training the emotion at stage 1 would contaminate acoustic features with textual ones since textual features are also relevant to the target task.

Many thanks for your detailed review!

[The proposed method assumes that we have labeled tasks for all potential downstream tasks. So one starts with general-purpose self-supervised representations...and ends up with representations that are (a) disentangled, but (b) are likely only useful for the tasks where we have labels. In this scenario, I am not entirely convinced that one has to use VIB]

We focus on scenarios where a speech model is fine-tuned for a downstream task, a widely used approach to adapt general-purpose models to specific applications. Our method leverages the same target task labels while incorporating an information bottleneck loss to achieve disentangled representations. Through our experiments (see Section 5), we show that using the Variational Information Bottleneck (VIB) effectively ensures that acoustic information is excluded from the textual representation and textual information is absent from the acoustic component.

Q6: [Do we actually need VIB? How different it would be if we used the labels to train a combination of ASR, Speaker and Emotion classifiers and used their outputs?]

We replicated our training procedure for Wav2Vec2-base to train the encoders without the information loss constraint and reported the results in Appendix (Figure H.1). Although this led to a slight improvement in task performance, it failed to disentangle the textual and acoustic information as intended (our evaluation in Section 5). Here, we can see textual features (red, Stage 1) are contaminated by acoustic features when we exclude information loss in training, which confirms the central role of the information loss term in our framework.

[If one needs to be confident that models do not use non-textual information while taking decisions, they can train models to make those decisions using pure transcripts]

Using pure transcription is not actually our goal. The accuracy of speech models is significantly better than that of text models for both emotion and speaker identification tasks, as both benefit much from acoustic features, and this is true in many other applications. Our goal is not to stop the model from using text, but rather to separate acoustic and textual features for controllability and interpretability; for example, we can see if a response of the dialog system is dependent on acoustic features or even reveal which layers and frames contribute textually or acoustically (as shown in the submission).