CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing

Thank you for your detailed comments, which have helped us improve our manuscript. Below are our clarifications:

Speaker Decoder and Generalization Ability: We respectfully disagree that the speaker decoder detracts from the results. We intend to show that integrating conditioning in the SSL improves unseen tasks. In Table 1, CA-SSLR conditioners/LID-decoder are trained on LID loss, then frozen, and the ASR decoder is trained on top. Thus, no component in the CA-SSLR encoder is trained for ASR but obtains improvements w.r.t. Baseline. In the same manner, CA-SSLR$^L$ models improve SV by 20% while the CA-SSLR$^L$ encoder conditioners were adapted on ASR and LID losses, not SV. Therefore, the speaker decoder is essential for demonstrating the generalization ability of our condition-aware approach and showing that CA-SSLR$^L$ is a generalist encoder.

Comparison with Houlsby Adapter ([1], Chen et al. [2023b]): CA-SSLR not only offers parameter-efficient fine-tuning (PEFT), but its principal contribution lies in incorporating dynamic adaptation to the language and speaker of the input, thereby improving generalizability, as noted by Reviewers iiaB and Jsg6. We performed additional experiments using the Houlsby adapter, as shown in Table A of the rebuttal PDF. Our ASR-CA-SSLR approach outperforms the ASR-Houlsby adapter, achieving WER of 18.6% and 31.6% compared to 20.3% and 34.6%, respectively. In SV, CA-SSLR improves EER from 1.29 to 1.15, while the Houlsby adapter worsens it from 1.29 to 1.37 compared to the XLS-R baseline. These results suggest that standard adaptation methods lack the enhanced generalization capabilities inherent to CA-SSLR. We appreciate the suggestions and plan to explore integrating our conditioner into the Houlsby adapter in future work.

Unbalanced Dataset and Performance It's important to highlight that ML-SUPERB includes 20 few-shot languages with only five utterances each for ASR to evaluate performance on unbalanced datasets. Improvement in few-shot CERs is documented in Tables 1 to 3, Figure 2, Appendix Table 8, and additional Table A. These results demonstrate CA-SSLR's robustness in unbalanced scenarios.

Fair Comparison and Hyperparameter Search: We confirm that the model parameters, learning rates, and batch size for ASR training are consistent with those in the ML-SUPERB benchmark paper. Parameters specific to the LID and SV decoders, such as dropout rates and hidden channels, were set using frozen SSLR settings to ensure a fair experimental setup. We added a reference in Section 4.2 to Appendix A.1 for clarification.

Enhancements in Presentation:

• Removed the duplicated content in Lines 121-131. This duplication was inadvertently introduced by one of the authors who didn’t notice that this content had been moved to another section.
• Revised the hierarchical writing with Figure A.
• Moved TCAC explanation before line 140 and substituted text with precise equations to assure reproducibility.
• Consolidated SV and LID configurations in lines 141-146 into a standard setup, relocated to the experimental section.
• Moved the information in 221-222 to the methods section.
• Included suggested parameter-efficient transfer learning references [1~4] in the related work and fixed the minor formatting issues.

We add the following equations for TCAC Implementation:

The TCAC module process latent representations at layer $l$, $\mathbf{S}^{(l)}\in\mathbb{R}^{C\times T}$, and the latest estimate of the conditioning features $\mathbf{z}\in\mathbb{R}^R$ and generates modulated latent representations $\mathbf{\tilde{S}}^{(l)}$ as

$\mathbf{\tilde{S}} _{t,c}^{(l)} = \text{TCAC}(S _{t,c}^{(l)}, \mathbf{z}) = \tilde{\gamma} _{t,c}^{(l)}(\mathbf{z}, S^{(l)})S _{t,c}^{(l)}+ \tilde{\beta} _{t,c}^{(l)} (\mathbf{z}, S^{(l)})$

Thus, the latent features are modulated by time-channel dependent scales $\tilde{\gamma} _{t,c}^{(l)}$ and biases $\tilde{\beta} _{t,c}^{(l)}$, obtained by:

$\tilde{\gamma}_{t,c}^{(l)} (\mathbf{z}, \mathbf{S}^{(l)})=\alpha_t^{(l)}(\mathbf{z}, \mathbf{S}^{(l)})\times \gamma_c^{(l)}(\mathbf{z})$

$\tilde{\beta}_{t,c}^{(l)} (\mathbf{z}, \mathbf{S}^{(l)})=\alpha_t^{(l)} (\mathbf{z}, \mathbf{S}^{(l)}) \times \beta_c^{(l)}(\mathbf{z})$

where channel-dependent $\gamma^{(l)},\beta^{(l)}\in \mathbb{R}^C$ are obtained as

$\gamma^{(l)}(\mathbf{z})=\mathbf{W}_\gamma^{(l)}\mathbf{z}+\mathbf{b} _\gamma^{(l)}$

$\beta^{(l)}(\mathbf{z}) = \mathbf{W}_\beta^{(l)} \mathbf{z} + \mathbf{b} _\beta^{(l)}$

The time-dependent scales $\alpha^{(l)} \in \mathbb{R}^T$ are obtained with an additive attention mechanism as

$\alpha^{(l)} _t(\mathbf{z}, \mathbf{S}^{(l)}) = \mathbf{v}^T _\alpha f(\mathbf{W} _\alpha^{(l)} [\mathbf{S} _t^{(l)T} \mathbf{z}^T]^T$+ $\mathbf{b} _a^{(l)})$

where $f(.)$ is a ReLU non-linearity, $\mathbf{W} _\alpha^{(l)}\in \mathbb{R}^{C'\times (C+R)}$, $\mathbf{b} _\alpha^{(l)}\in\mathbb{R}^{C'}$, and $\mathbf{v _\alpha}\in\mathbb{R}^{C'}$.

We obtained the conditioning features $\mathbf{z}$ from the hard decisions or internal embedding layer $\mathbf{e}\in\mathbb{R}^E$ of the intermediate decoders, by $\mathbf{z} = \mathrm{LayerNorm}(\mathbf{W} \mathbf{e} + \mathbf{b})$, where the affine transform parameters $\mathbf{W}$, $\mathbf{b}$ are shared across TCAC layers.

Additional Clarifications:

• Alpha, beta, and gamma are functions that predict vectors, in parts of the text the dependency on (z,S) is dropped to keep the notation uncluttered.
• The conditioning feature can be derived from the decoder's second last layer (embedding layer) or decoder output (either as hard or soft labels).
• The LID decoder was updated with the pre-trained, frozen SSLR model but remained frozen during TCAC training, as shown in Figure A.
• "ASR-adapted" refers to training CC/TCAC with ASR loss only.
• TCAC is integrated within wav2vec 2.0 layers after the self-attention module. These details are included in the model architecture section.

Thanks for the comment. Here are some clarifications.

Regarding the generalist model concern, we want to emphasize that CA-SSLR is considered a generalist model because it maintains the base model's integrity while improving performance on previously unseen tasks. In Table 1, CA-SSLR conditioners/LID-decoder are trained on LID loss, then frozen, and the ASR decoder is trained on top. Thus, no component in the CA-SSLR encoder is trained for ASR but obtains improvements w.r.t. Baseline. In the same manner, CA-SSLR$^L$ models improve SV by 20% while the CA-SSLR$^L$ encoder conditioners were adapted on ASR and LID losses, not SV, showing that CA-SSLR$^L$ is a generalist encoder.

We acknowledge the concern about using the term "Time-wise Attention." Although it seems like time-dependent scaling, it fits the concept of additive attention, without normalizing weights to sum up to one with softmax. We observed that removing the softmax provided better results. Our approach involves using a linear projection to compute $\alpha _t^{(l)}(z, S^{(l)} _{t,c}) \in \mathbb{R}^T$ by concatenating a single feature vector $\mathbf{z}$ with $S^{(l)} _{t,c}$ at each timestep, similar to the method used in [A]. This process assigns different weights to each timestep, allowing us to scale $S^{(l)} _{t,c}$ accordingly. Even though there are no direct interactions between timesteps, the dynamic assignment of weights based on features aligns with the principles of attention, as the approach applied in [B].

In Table 4, we have updated CA-SSLR$^{L,S}$ TCAC results for comparison with XLS-R, achieving the best results in LID and ASR and ranking second in SV, showing strong adaptation capabilities while maintaining competitive generalization ability.

Regarding the confusion about frozen and trainable parameters, first, we train the LID decoder on the pre-trained SSL model. The decoder gets a weighted average of the SSL encoder layers. Afterward, this LID decoder is then frozen and used to get the conditioning features that are fed to the LID TCAC conditioners. In this case the LID decoder gets a weighted average of the CA-SSLR encoder layers computed up to that point, and the conditioning feature is recomputed every three CA-SSLR layers. Here, we only train the linear projection before the LID decoder and the LID TCAC parameters to the LID-conditioned CA-SSLR encoder, termed CA-SSLR$^L$. Similarly, the SV decoder is trained on top of the CA-SSLR$^L$ then frozen. Then, we just train the SV TCAC parameters to get the CA-SSLR$^{L,S}$. The adaptation training has been detailed in Figure A of the rebuttal PDF and will be added to Sec. 3. We also update lines 144 and 145 to avoid confusion.

Regarding parameter sharing, the only layer-dependent features used for calculating the condition feature z are the linear projections for the decoders and the weights used for the weighted sum of the SSL layers before the linear predictions, as shown in Figure A of the rebuttal PDF. All other decoder parameters are shared.

For the extended few-shot condition, the original ML-SUPERB settings include "normal languages" with 10 minutes to 1 hour of data per language and "few-shot languages" with only 5 utterances each. In the "extended few-shot condition," we incorporate the language labels from these few-shot data for LID training but continue using only 5 utterances with transcriptions for ASR training. This is because language labels are more accessible to obtain than transcriptions. We will add this table to the appendix to clarify these settings.

Regarding whether the "pre-trained and fixed SSLR model" refers to XLS-R, Yes, it does refer to the frozen pre-trained XLS-R model with a trained decoder head. We will update the sentence to "frozen XLS-R model with ASR decoder" for clarity.

Regarding the condition feature z, we have updated the sentence in L163 for clarity: "We re-estimate the condition feature 𝑧 from the updated language or speaker embedding every three layers using a linear projection and layer normalization, which are shared across layers.” This has been revised to “As shown in Figure A, 𝑧 is derived from the updated language or speaker embedding through a linear projection layer or an embedding layer for hard LID/SV labels.”

[A] Luong, M. T., Pham, H., & Manning, C. D. (2015, September). Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 1412-1421).

[B]Lin, Z., Feng, M., dos Santos, C. N., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2022, July). A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING. In International Conference on Learning Representations.

Thank you for the insightful comment. Here are some clarifications.

Regarding inference cost, the encoder parameters are shared among all three tasks, which helps to minimize expenses. The LID decoder is lightweight, with an RTF of less than 0.001, so it doesn't significantly add to the computational load of the ASR task. While the ASR and SSL models are more complex, with single-pass RTFs of 0.004 and 0.016, respectively, they may need multiple inferences based on the decoding algorithm. As shown in Tables 9 and 10, the CA-SSLR system achieves 30 to 45% faster RTF than multi-task baselines across different components and scenarios.

Concerning the cascaded baseline, we examined the cascading baseline for LID + ASR in Table 2 and Figure 2. In this method, we train the LID decoder with a pre-trained SSL model and use another SSL model with TCAC components to train the ASR decoder. Our findings indicate that the TCAC can achieve performance comparable to a fully fine-tuned approach for languages with abundant resources and surpasses the fully fine-tuned approach in few-shot ASR languages.

To assess training speed and peak memory usage, we compared the CA-SSLR approach with the additional baseline, Houlsby Adapter, and a fully fine-tuning approach (the results comparing the Houlsby Adapter and CA-SSLR$^{L}$ are detailed in Table A of the rebuttal PDF). We evaluate the training speech for 10k iterations with batch size 8, and the results are shown in the following table:

we found that the CA-SSLR approach ranks second compared to the Houlsby Adapter and a fully fine-tuning approach in speed and memory usage. However, CA-SSLR surpasses the Houlsby Adapter in adaptation effectiveness, as shown in Table A(b) of the rebuttal PDF. It also demonstrates the best generalization ability among the three methods, detailed in Tables 1, 4, and A. Additionally, we acknowledge that the current implementation is not yet optimal and are committed to further improvements.

Thank you for the helpful comments. Here are some clarifications.

First, for the generalist model without fine-tuning, the SOTA results for pre-trained SSLR models can be found in the paper ML-SUPERB challenge [A], where the MMS-1b model performs the best with 1hr ASR WER 18.1% and LID accuracy 86.1% respectively (see the table below). However, we used XLS-R 300M to make our experiments feasible.

Second, with fine-tuning, the single-task SOTA results can be found in Table 1 and Figure 2, with 90.1% LID accuracy and 17.3% ASR CERs.

We found that compared with the MMS-1b baseline, the proposed system achieves better results in Normal languages and comparable results in few-shots languages than the model with three times more parameters (300M vs. 1B). We will add these results in Table 3.

Regarding the concern about using shallow layers, we would like to clarify that the CA-SSLR system recomputes the embeddings every 3 or 4 layers rather than predicting only once in the initial layers, as shown in Table A in the rebuttal PDF. From the SSL feature weights utilized by the LID and SV decoders, we observe that LID weights are evenly distributed, while SV weights are more concentrated in the earlier layers. Consequently, incorporating predictions from higher LID layers can enhance LID accuracy and improve conditioning features. Conversely, for SV, using only the first 6 or 12 layers of XLS-R might be sufficient. We will further explore this in future work.

While CC applies the same transformation to all time frames, we added the attention module in TCAC to be able to emphasize different time-frames of the encoded features within the SSLR layers. When deciding between TCAC and CC, our experiments indicate that TCAC provides better results for LID and ASR. However, as shown in Table 4, CC currently yields slightly better results for SV. Overall, relative improvements of TCAC in ASR and LID are larger than relative degradation in SV. We plan to conduct further investigations to enhance the SV performance of TCAC. In the updated version, we will provide visualizations of the attention mechanism for the TCAC module using one or two examples. We will further provide a more detailed analysis of our results in the revised version.

[A] Shi, J., Chen, W., Berrebbi, D., Wang, H. H., Huang, W. P., Hu, E. P., ... & Watanabe, S. (2023, December). Findings of the 2023 ml-superb challenge: Pre-training and evaluation over more languages and beyond. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 1-8). IEEE.

Thanks for the insightful comments. We have included additional equations and figures to improve the clarity and address the writing in Sec. 3.

Regarding additional figures, we add Figure A in the rebuttal PDF to clarify our system as the elaboration for the original Figure 1.

Regarding additional equations, we further add details equations to clarify the Time-Channel Attention Conditioner to form a separate subsection of Sec 3.2 to improve the readability:

"As depicted in Fig.1b, the TCAC module ingests the latent representations of the CA-SSLR at layer $l$, $\mathbf{S}^{(l)}\in\mathbb{R}^{C\times T}$, and the latest estimate of the conditioning features $\mathbf{z}\in\mathbb{R}^R$ and generates modulated latent representations $\mathbf{\tilde{S}}^{(l)}$ as

$\mathbf{\tilde{S}} _{t,c}^{(l)} = \text{TCAC}(S _{t,c}^{(l)}, \mathbf{z}) = \tilde{\gamma} _{t,c}^{(l)}(\mathbf{z}, S^{(l)})S _{t,c}^{(l)}+ \tilde{\beta} _{t,c}^{(l)} (\mathbf{z}, S^{(l)})$

where $t$, $c$, and $l$ represent time, channel, and layer indices, respectively. Thus, the latent features are modulated by time-channel dependent scales $\tilde{\gamma} _{t,c}^{(l)}$ and biases $\tilde{\beta} _{t,c}^{(l)}$. These are obtained as the products:

$\tilde{\gamma} _{t,c}^{(l)} (\mathbf{z}, \mathbf{S}^{(l)})=\alpha _t^{(l)}(\mathbf{z}, \mathbf{S}^{(l)})\times \gamma _c^{(l)}(\mathbf{z})$

$\tilde{\beta}_{t,c}^{(l)} (\mathbf{z}, \mathbf{S}^{(l)})=\alpha_t^{(l)} (\mathbf{z}, \mathbf{S}^{(l)}) \times \beta_c^{(l)}(\mathbf{z})$

where channel-dependent $\gamma^{(l)},\beta^{(l)}\in \mathbb{R}^C$ are obtained as

$\gamma^{(l)}(\mathbf{z})=\mathbf{W}_\gamma^{(l)}\mathbf{z}+\mathbf{b} _\gamma^{(l)}$

$\beta^{(l)}(\mathbf{z}) = \mathbf{W}_\beta^{(l)} \mathbf{z} + \mathbf{b} _\beta^{(l)}$

The time-dependent scales $\alpha^{(l)} \in \mathbb{R}^T$ are obtained with an additive attention mechanism as

$\alpha^{(l)} _t(\mathbf{z}, \mathbf{S}^{(l)}) = \mathbf{v}^T _\alpha f(\mathbf{W} _\alpha^{(l)} [\mathbf{S} _t^{(l)T} \mathbf{z}^T]^T$+ $\mathbf{b} _a^{(l)})$

where $f(.)$ is a ReLU non-linearity, $\mathbf{W} _\alpha^{(l)}\in \mathbb{R}^{C'\times (C+R)}$, $\mathbf{b} _\alpha^{(l)}\in\mathbb{R}^{C'}$, and $\mathbf{v _\alpha}\in\mathbb{R}^{C'}$.

We obtained the conditioning features $\mathbf{z}$ from the hard decisions or internal layer (embedding layer) $\mathbf{e}\in\mathbb{R}^E$ of the intermediate decoders, by $\mathbf{z} = \mathrm{LayerNorm}(\mathbf{W} \mathbf{e} + \mathbf{b})$, where the affine transform parameters $\mathbf{W}$, $\mathbf{b}$ are shared across TCAC layers.

Thus, TCAC enables the model to dynamically adjust its behavior to the input audio in response to the provided conditioning features."

Also, for the Condition-Aware Learning Mechanism, we remove Lines 121-131 to eliminate duplication and focus the section on our unique contribution. This duplication was inadvertently introduced by one of the authors who didn’t notice that this content had been moved to another section. Following are the revisions of the second and third paragraphs in Sec. 3.1:

“The proposed CA-SSLR model improves the efficiency of evaluating multiple tasks by introducing a hierarchy of conditioners within the pre-trained SSL encoder layers. It utilizes intermediate predictions from the language identification (LID) and speaker verification (SV) decoders as conditions to recursively adapt subsequent layers, as shown in Figure A. This hierarchical approach structures the SSL encoder layers so that each layer refines its output based on the predictions of the preceding layers. Early layers produce embeddings that capture the essential language and speaker characteristics, which are then used to inform scaling and bias adjustments in later layers.

We propose a novel mechanism, the Time-Channel Attention Conditioner (TCAC), which modulates the encoder's hidden representations using time-channel-dependent scales and biases. This approach enables the SSL encoder to dynamically adjust to varying tasks and input conditions. The inputs to these conditioners are embeddings derived from intermediate evaluations of the LID and SV decoders."