Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments

Q1: Line 175: For the entire "The Encoder Fusion Module" section, this is the most important part, but it is a bit difficult to follow. I would suggest using equations wherever possible. Also, consider adding a new diagram explaining how this module works. Line 173: I assume T_pos is number of frames of positive enrollment audio? Worth clarifying.

A1: Thank you for your suggestions. We provide further explanation on the Fusion module in the response to reviewer iQ1f Q5, and will include the pseudo-code of the module in the revised version. T_pos indeed represents the number of STFT frames in the Positive Enrollment.

Q2: "Model performance under inaccurate Positive and Negative Labeling" - it is really great to have this section. But I think it can be improved. First, 0.30 ± 0.19 seconds do not seem to be super inaccurate. What if the speaker boundary errors are above 1 second? Also, I am not sure I understand Figure 6. What is x-axis? Each column is one sample? Why the red square can be sometimes high (sample 7) and sometimes low (sample 10)? Please consider remaking this Figure or replace with a table.

A2: Thank you for your suggestions. In the user study on inaccurate positive and negative labeling, among the 100 user-provided labels across 10 samples, 18 labels for audio samples [1, 3, 4, 6, 10] deviate more than one second from the ground truth boundaries. We report the average model SI-SNRi on these inaccurate labels in the table below. Sample [1,3,4,6] all show less than 1 dB SI-SNRi performance degradation compared to the model performance using ground truth boundaries, despite a temporal labeling inaccuracy of 1.36 $\pm$ 0.44 seconds. For sample 10, only one user label deviates by more than one second, reaching 4.1 seconds. This results in 5.16 dB SI-SNRi. Overall, these results demonstrate the robustness of our model to inaccuracies in user-provided boundary annotations.

Regarding Figure 6, you are correct that the x-axis represents the sample ID, and each column corresponds to the model performance on an individual sample. The red markers indicate outliers, which appear above or below the box plot when a model’s performance on one of the 10 user labelings for that sample exceeds the upper quartile or falls below the lower quartile, respectively. To improve clarity, we will revise Figure 6 as a table reporting the mean and standard deviation of model performance for each sample.

Q3: All the experiments are based on synthetic datasets. Although it is common, it's not ideal, especially this paper aims to addess a special use case in real world applications.

A3: Thank you for mentioning this. Indeed, to effectively demonstrate our model's performance under strong overlap between the target and interfering speakers, all audio mixtures used in our original submission were synthetically generated or manually added up from real-world audio recordings. We additionally compare our model and the baseline methods on naturally recorded real-world audio mixtures. Please refer to the response to reviewer 1MM1 Q1 for the result.

Q4: What is the mechanism to remove Positive Interferer (PI) in the Encoder Fusion Module? This part is very unclear to me. Both the model architecture or the loss function of stage 1 do not seem to have a built-in mechanism to remove positive interferer. I'd like to see more clarification or explanation on this.

A4: Sorry for the confusion. We based the model Encoder Fusion Module on self-attention layers. The self-attention operation allows model to perform pairwise comparison between the embeddings of different frames in the Positive Enrollment. As a result, this architecture design allows the model to identify the speakers who do not talk in some of the Positive Enrollment frames and exclude them in the extracted embedding. To guide the model exclude PI's characteristic in its embedding, we simulated the Positive Enrollments so that the number of PI varies between 0 to 2 in the stage 1 training. Distilling the encoder to only encode the target speaker's voice in the Positive Enrollment forces the model to exclude the PI's voice characteristics in its extraction. Please see the response to reviewer iQ1f Q5 for more details.

Sorry for the confusion, we provide additional clarification on the Encoder Fusion module:

Q1: How are the segmentation embeddings initialized? Do you have any loss function to penalize these segmentation embeddings to become too similar?

A1: We realize that the term "segmentation embedding" used in our paper refers to what is commonly known as "segment embedding" in Transformer-based models such as BERT. We apologize for the terminological inconsistency and will revise the terminology accordingly in the final version to avoid confusion.

The segment embeddings in our model are randomly initialized and jointly trained with other learnable parameters in the Encoding Branch. There is no additional loss term penalizing the similarity between the segment embeddings of the Positive and Negative Enrollments. This design choice follows the standard practice in BERT-based models[1, 2, 3], which also train randomly initialized segment embeddings without explicit constraints.

Q2: What do you mean by full-band, and why does the Encoder Fusion Module have to be full-band?

A2: As noted in line 161 of the original paper, the term "Full-band Self-attention" originates from the prior work TFGridnet[4]. Since this module was directly adopted from the prior work, we simplified the explanation regarding the shapes of the positive and negative enrollment embeddings. We provide those details here for clarity.

For simplicity, we referred to the Positive and Negative Enrollment embeddings as having shape $[T_{\text{pos}}, D]$ and $[T_{\text{neg}}, D]$. Their actual shapes are $[T_{\text{pos}}, F, E]$ and $[T_{\text{neg}}, F, E]$, respectively, where $T_{\text{pos}}$ and $T_{\text{neg}}$ denote the number of time frames, $F$ is the number of frequency bins, and $E$ is the embedding dimension for each time-frequency unit. After the elementwise-addition with the $[F, E]$ shaped segment embeddings and concatenation, the input to the Full-band Self-attention module has shape $[T_{\text{pos}} + T_{\text{neg}}, F, E]$.

The Full-band Self-attention module flattens the embedding and frequency dimension, resulting in a tensor of shape $[T_{\text{neg}} + T_{\text{pos}}, F * E]$ before applying self-attention. The term Full-band reflects that the $E*F$ dimensional embedding vector of each frame encodes all the frequency bands, allowing the attention mechanism to operate over full spectral information.

We adopt this module in the Encoder Fusion Module because the task of fusing positive and negative enrollment embeddings requires discriminating between different speakers' characteristics, which are distributed across a wide frequency range. A module that integrates information from the entire frequency range is therefore better suited to capture speaker-specific cues that might lie in different frequency bands.

We apologize again for the unclear explanation and the inconsistent terminology, and hope the above clarification resolves your questions. We are happy to provide further clarification if there are any remaining concerns.

References:

[1] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A Lite BERT for self‑supervised learning of language representations,” in Proc. International Conference on Learning Representations (ICLR), 2020.

[2] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai, “VL‑BERT: Pre‑training of Generic Visual‑Linguistic Representations,” in Proc. International Conference on Learning Representations (ICLR), 2020.

[3] J. Devlin, M.‑W. Chang, K. Lee, and K. Toutanova, “BERT: Pre‑training of deep bidirectional transformers for language understanding,” in Proc. NAACL‑HLT, 2019, pp. 4171–4186.

[4] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “TF‑GridNet: Integrating Full‑ and Sub‑Band Modeling for Speech Separation,” IEEE/ACM Trans. Audio, Speech & Lang. Process., 2023.

Q1: The experimental results are mostly relying on artificially mixed audio data. Given that this paper aims to address the limitation of prior works for real-world TSE applications, it is expected to see more evaluation results of the method on naturally recorded mixture audio, to further confirm its effectiveness. Although there are some relevant results presented in Appendix K, it seems that the mixture signals in the experiment are still manually added up. In addition, there is no comparison with other existing approaches.

A1: Thank you for raising this question. To effectively demonstrate our model's performance under strong overlap between the target and interfering speakers, all audio mixtures used in our original submission were synthetically generated or manually added up from real-world audio recordings.

We additionally compare our model and the baseline methods on naturally recorded real-world audio mixtures. We evaluate the Mean Opinion Score (MOS) of different models using five naturally recorded audio mixtures sourced from Freesound.org and the VoxConverse dataset. These sounds are noisy real-world speech recordings captured in pubs, metro stations, urban areas, and city council meetings. For the evaluation, we manually labelled the Positive and Negative Enrollment and extracted target speech from Audio Mixture clips taken from the same recording. The duration of the enrollment and Audio Mixture clips range from 3 to 8 seconds.

10 participants are asked to rate the original (unprocessed) Audio Mixture, and the audio extracted by three models. Given a textual description of the target speaker (e.g., the female speaker talking over the crowd), they rated the clarity of the target speech relative to background noise on a 1–4 scale. As shown in the table below, our model achieved a MOS of 3.35, outperforming all baseline methods. These results demonstrate the superiority and practical applicability of our model in real-world target speech extraction scenarios.

Q2: The loss function for training the extraction branch, i.e., eq. (3) seems relatively simple -- it is just the SNR between the ground truth signal and the output of the TSE model. I wonder if this would lead to sub-optimal performance and what motivated you to use this loss? Also, have you experimented with other more sophisticated loss functions, e.g., time-frequency domain MSE losses or some other perceptually motivated losses?

A2: Thank you for sharing your insight. Training using only SNR loss is a common practice used in numerous prior works [1, 2, 3, 4]. We kept the loss simple and consistent with the prior works to improve the reproducibility and comparability of our work. Therefore, we did not conduct extensive ablation studies on alternative loss functions. Incorporating perceptual losses is indeed a valuable direction for future work to enhance model performance.

[1] J. Huang, X. Wang, and D. Wang, “Multi-level speaker representation for target speaker extraction,” in Proc. ISCA Interspeech, 2021, pp. 1459–1463.

[2] A. Ephrat, I. Mosseri, T. M. Remez, and S. Freeman, “Target conversation extraction: Source separation using turn-taking dynamics,” in Proc. ISCA Interspeech, 2020, pp. 1401–1405.

[3] Y. Xu, M. Yu, C. Yu, and D. Yu, “Listen to extract: Onset-prompted target speaker extraction,” in Proc. ISCA Interspeech, 2022, pp. 2963–2967.

[4] W. Zhou, S. Chen, Y. Liu, and D. Su, “Look once to hear: Target speech hearing with noisy examples,” in Proc. ISCA Interspeech, 2023, pp. 3597–3601.

Q3: Will you make the code public if the paper is accepted?

A3: Yes. The code and checkpoints will be made public once accepted.

Q1: I think it is important to report performance when a clean enrollment is available. Without this, we cannot judge whether the noisy strategy matches or degrades relative to the ideal case.

A1: We reported our model performance when clean enrollments are available in Appendix J in the original supplementary material. Our model outperforms the TSE model trained on clean enrollment when extracting from the mixtures of three or more speakers, but shows lower performance when extracting from the mixtures of two speakers. In addition, we evaluate our model and the baseline models' performance when using noisy single speaker enrollments. Please see the response to reviewer anep Q2 for the result.

Q2: Including recent SOTA discriminative and generative TSE methods (e.g. USEF-TSE, SoloAudio) would give a fairer picture of the gains.

A2: Thank you for your suggestion. We did not focus our comparison with the SOTA TCE methods because of the difference in problem formulation. Since the enrollments contain multiple speakers, SOTA TSE models trained on clean enrollment cannot perform accurate extraction without additional information to distinguish target speakers from the interfering speakers in the noisy enrollment.

To verify this, we additionally evaluate the USEF-TFGridnet and SoloAudio under the proposed task. As shown in the table below, both model performance degrade significantly under our problem formulation, highlighting the increased difficulty of our task and demonstrating the effectiveness of our proposed model. In particular, we notice that the SoloAudio model trained to perform target sound extraction struggles to distinguish the subtle difference between different speakers' characteristics. As a result, it often produces outputs with large silent regions, leading to notably low SI-SNRi scores.

Q3: The current metrics (SNR, SI-SNR, STOI, DNSMOS) omit key measures of intelligibility (WER or CER) and perceptual quality (e.g. PESQ). I think adding these would strengthen the assessment.

A3: Thank you for your suggestion on additional metrics. We include further evaluations of our models and baseline methods using PESQ and WER in the table below. Our monaural model outperforms all the baseline methods in both metrics. However, as mentioned in the Conclusion and Limitation section, the generated audio still contains artifacts, which negatively impact the PESQ and WER performance. Since the binaural model is trained to preserve the reverberation characteristics of the target speaker, and reverberation can negatively impact PESQ and WER, we report the binaural models' results here solely for completeness.

Q4: Objective gains do not always reflect perceptual quality. I would like to see human listening tests or a comparison demo to confirm that improvements hold up for end users.

A4: We provided five audio examples each for the successful and failure cases of our model in the supplementary materials.

Additionally, we evaluate the Mean Opinion Score (MOS) of different models using naturally recorded audio mixtures sourced from Freesound.org and the VoxConverse dataset. These sounds are real-world audio mixtures captured in pubs, metro stations, urban areas, and city council meeting. Please see the response to reviewer 1MM1 Q1 for the results.

Q5: Some notation in Section 3.3 (encoder fusion) is dense. A short pseudo-code snippet or an extra diagram could help readers grasp the self-attention comparison more easily.

A5: Thank you for your suggestion. We provide the pseudo-code of the Encoder Fusion Module below. Two segmentation embeddings, $S_{pos}, S_{neg}$, are first element-wise added to the input embeddings $E_{pos}, E_{neg}$ to allow the model to distinguish which enrollment each embedding originates from. The resulting two embeddings are concatenated along the temporal dimension and passed through two Full-band Self-attention layers.

The self-attention calculation between the embeddings of different Positive Enrollment frames allows the model to identify the Positive Interferers, who remain silent in some of the Positive Enrollment frames, and exclude these speakers in the extracted embedding. Similarly, attention between the Positive and Negative Enrollments embeddings enables the model to identify and exclude Negative Interferers, (i.e., speakers present only in the negative enrollments), from the output embedding.

During Stage 1 training, we simulate the enrollments such that the number of Positive and Negative Interferers varies between 0 and 2. Distilling the encoder to produce an embedding that retains only the target speaker’s characteristics under these enrollments input encourages the model to exclude both types of interferers in the extracted speaker embedding.

# Input: 
#     E_pos: Positive Enrollment embedding, shape [T_pos, D]
#     E_neg: Negative Enrollment embedding, shape [T_neg, D]
# Learnable parameters: 
#     S_pos: segmentation embedding for E_pos, shape [1, D]
#     S_neg segmentation embedding for E_neg, shape [1, D]
#     M: 2 Full-band Self-attention layers
Encoder Fusion Module:
  # Elementwise add segmentation embedding via broadcasting
  E_pos = E_pos + S_pos # shape [T_pos, D]
  E_neg = E_neg + S_neg # shape [T_neg, D]

  # Concatenate along the temporal dimension 
  E_concat = [E_pos, E_neg] # shape [T_pos + T_neg, D]

  # Apply two Full-band Self-attention calculation
  E_concat = M(E_concat) # shape [T_pos + T_neg, D]

  # Truncate embeddings to match the teacher model's embedding shape
  output = E_concat[:T_pos] # shape [T_pos, D]

Q1: When the input signal is binaural (as described in Section 4.1), it is unclear whether the proposed model is capable of modeling the spatial information between two microphone channels, as Section 3 only discusses monaural scenarios.

A1: Thank you for raising this question. The difference between the monaural and binaural audio architectures lies solely in the input. The binaural model takes in 4 channels input (stacked real and imaginary components from both channels) and the monaural model takes in only 2 channels (stacked real and imaginary component). By stacking the two channels' STFT, the binaural model can capture the inter-channel temporal differences between the two channels, which encode the directional information of the target speaker since the target speaker speaks at 90 degree azimuth angle in the enrollment. Both model architectures do not differ after the first convolution layer.

To verify that our model leverages spatial information for speaker extraction, we evaluate the binaural model by randomly varying the azimuth angle of the target Speaker in the Positive Enrollment. As shown in the table below, this results in a significant drop in performance. This indicates that our binaural model leverages the spatial information (target speaker direction) to perform enrollment.

Q2: Model performance with single-speaker noisy enrollments.

A2: Thank you for your suggestion on additional experiments. We re-evaluate our model and two baseline models using noisy single speaker enrollments. The noise in the enrollment is from the WHAM! dataset and set at 0 SNR level. The table below shows the model performance when extracting from the mixture of two or three speakers, TSE models trained with clean enrollments show significant performance decrease under noisy single speaker enrollments. In comparison, our model consistently outperforms the baselines across all metrics except DNSMOS, demonstrating greater robustness to enrollment noise.

Q3: Line 323: Explanation of "correctly extracts the target speaker" and analysis of the target speaker confusion problem.

A3： Thank you for your suggestion. By "correctly extracts the target speaker", we mean the extracted audio achieving a higher SI-SNR with respect to the target speaker's voice than with respect to the interferer's speech. If the model mistakenly encodes the interfering speaker in the enrollment, it should instead show a high SI-SNR for the interfering speaker. As shown in Figures 4 and 5 in the original paper, the extracted speech has less than -10 Si-SNRi with respect to the interfering speaker's speech, verifying that the interfering speakers' voice is effectively suppressed.

To further investigate whether our model suffers from the target speaker confusion problem, we follow the experimental setup proposed by Zhao et al. [1]. Specifically, we construct 5000 test samples where both the audio mixture and the enrollment contain two speakers, and share the same interfering speaker. Let $P_{pred}$ be the enrollment pair in the sample, where speaker A is the target speaker and speaker B is the interfering speaker. After performing extraction, we identify the enrollment pairs that result in the extracted audio being more similar (in terms of SNR) to the interfering speaker B's speech than to the target speaker A's. We refer to these samples as target confusion samples. For each target confusion sample, we construct two additional enrollment pairs: $P_{tgt}$ and $P_{interfer}$. In $P_{tgt}$, same as the target confusion sample, speaker A is the target and speaker B is the interferer. In $P_{interfer}$, speaker B is the target and speaker A is the interferer. The table below summarises the role of speaker A and speaker B in the audio mixture and each enrollment pair.

To verify if the model mistakenly encode the interfering speaker as the target speaker and the interfering speaker, we flatten and normalize the embeddings extracted by our model from each enrollment pair, obtaining $E(P_{pred})$, $E(P_{tgt}), E(P_{interfer})$, and compute the cosine similarity between $E(P_{pred})$ and $E(P_{tgt})$, as well as between $E(P_{pred})$ and $E(P_{interfer})$.

Out of the 5000 tested samples, 74 of them are the target confusion samples. 34 samples out of the 74 target confusion samples (45.9%) have enrollment embedding closer to the interfering speaker than to the target speaker (i.e. 34 of the samples have $cos(E(P_{pred}), E(P_{tgt})) < cos(E(P_{pred}), E(P_{interfer}))$). These findings are consistent with those reported in the target speaker confusion study [1], where 45.1% of target confusion samples showed the extracted embedding being closer to the interferer. This means that significant percentage of target confusion samples have encoder embeddings being more close to the interfering speaker. However, it is important to emphasise that only 74 out of the 5000 samples (1.48%) showed evidence of target speaker confusion in our evaluation, suggesting that this is a rare occurrence and does not pose a significant issue for our model's overall performance.

[1] Z. Zhao, D. Yang, R. Gu, H. Zhang, and Y. Zou, “Target confusion in end-to-end speaker extraction: Analysis and approaches,” in Proc. ISCA Interspeech, 2022, pp. 5333–5337.

Q4: Additional evaluation metrics and baseline comparison on real-world speech data.

A4: We additionally evaluate our model with WER and PESQ on the synthetic data. Please see the response to reviewer iQ1f Q3 for the results. In addition, since we do not have access to the ground truth of the extracted audio in audio mixtures recorded in the real-world, we focus our evaluation on the Mean Opinion Score for the real-world speech data. Please see the response to reviewer 1MM1 Q1 for the results.

Q5: The artefacts mentioned in line 365 cannot be easily understood without an example.

A5: We provided 5 audio examples for each of successful and failure cases of our model in the original supplementary file. We will implement a demo page to present the extraction results on both synthetic and real-world audio mixtures.

Q6: More advanced TSE baselines, such as X-TF-GridNet, instead of the outdated SpeakerBeam model, should be compared in the experiments to better assess the performance of the proposed method.

A6: Thank you for your suggestion. As USEF-TFGridnet[2] reports higher performance in comparison to the X-TFGridnet in their paper, we compare with USEF-TFGridnet and SoloAudio. Please see the response to reviewer iQ1f Q2 for the results.

[2] B. Zeng and M. Li, “USEF‑TSE: Universal Speaker Embedding Free Target Speaker Extraction,” IEEE Trans. Audio, Speech and Language Processing, 2025, pp. 2110–2124.

Paper Formatting Concerns and Clarifying Technical Details Thank you for raising these concerns. We will refine the grammar and explanation in the corresponding paragraphs in the revised version. In particular, we will include pseudo-code of the Encoder Fusion module in the response to reviewer iQ1f Q5. We will also discuss the potential negative social impact mentioned.