Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder

We thank the reviewer for the detailed feedback and constructive questions. We are encouraged by the recognition of the strengths of our work.

Below are the detailed responses to each query.

Weaknesses

The novelty is limited. As introduced in Section 2, multi-layer feature learning and joint training of speech tasks have been investigated in plenty of previous studies. The main contribution of this paper is to include more speech tasks into joint training and to combine it with multi-layer feature learning, which is limited in my opinion.

We acknowledge the reviewer’s comment regarding the lack of novelty in the multi-layer feature fusion/weighted sum approach. As noted in Section 2.1, our contribution lies in leveraging the weighted sum of OWSM encoder features to develop a unified speech model capable of addressing diarization, separation, and multi-speaker ASR tasks simultaneously. Integrating these tasks into a single multi-task framework presents a significant challenge due to their inherently disjoint nature. Our work addresses this issue by developing a unified framework that aligns these tasks through shared representations while allowing task-specific adaptations. This approach not only reduces redundancy in task-specific models but also promotes mutual benefits, such as improved speaker attributed transcription and better signal separation for diarization.

---

As shown in experimental results, the task weights for joint training in Eq.(16) play an important role in model performance. Thus, how to optimize these weights should be carefully explained.

We adopted a weighted-sum scalarization approach [1] to simplify the multi-objective optimization problem into a single-objective [2] one by assigning equal weights to all task-specific losses. This approach assumes that the tasks are cooperative rather than conflicting, particularly in our two-speaker and three-speaker scenarios, and reflects their equal importance in our framework. Since the primary goal of this study is to develop a unified framework capable of integrating multiple tasks rather than optimizing individual task performance, we propose an equal-weighting strategy that assigns equal importance to all tasks. This approach is validated by experimental results, which demonstrate that simple equally weighted scalarization achieves state-of-the-art performance.

We have added the above discussion in Section 4.3.

---

Questions

According to the results shown in Table 3 and 4, the layer weights for the three tasks were quite similar in both joint training configurations. Therefore, it may not be necessary to use task-dependent layer weights. How about the performance of using unifed layer weights across different tasks?

We will attempt to do the experiments with the unified layer weights. However, fully completing the experiments might not be feasible within the rebuttal timeline. We will do our best to include preliminary results during the revision phase.

References

[1] Matthias Ehrgott. Weighted Sum Scalarization, pp. 55–75. Springer Berlin Heidelberg, Berlin,Heidelberg, 2000. ISBN 978-3-662-22199-0. 
[2] Cristina Bazgan, Stefan Ruzika, Clemens Thielen, and Daniel Vanderpooten. The power of the weighted sum scalarization for approximating multiobjective optimization problems. Theory of Computing Systems, 66(1):395–415, Feb 2022. ISSN 1433-0490. doi: 10.1007/s00224-021-10066-5.

We thank the reviewer for the detailed feedback and constructive questions. We are encouraged by the recognition of the strengths of our work.

Below are the detailed responses to each query.

Weaknesses

The manuscript lacks novelty. Numerous works have already discussed the use of multi-layer feature fusion/weighted sum, such as "Large-Scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification," "Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers," and "Resource-Efficient Transfer Learning from Speech Foundation Model Using Hierarchical Feature Fusion." Compared to the aforementioned studies, this manuscript makes no novel contributions to methodology. Instead, it leverages existing techniques to address three speech-related tasks in multi-speaker scenarios.

We acknowledge the reviewer’s comment regarding the lack of novelty in the multi-layer feature fusion/weighted sum approach. As noted in Section 2.1, our contribution lies in leveraging the weighted sum of OWSM encoder features to develop a unified speech model capable of addressing diarization, separation, and multi-speaker ASR tasks simultaneously. Integrating these tasks into a single multi-task framework presents a significant challenge due to their inherently disjoint nature. Our work addresses this issue by developing a unified framework that aligns these tasks through shared representations while allowing task-specific adaptations. This approach not only reduces redundancy in task-specific models but also promotes mutual benefits, such as improved speaker attributed transcription and better signal separation for diarization.

---

The authors overclaimed achieving state-of-the-art performance in speaker diarization, speech separation, and multi-speaker ASR. However, for speech diarization, they only compared their method with ConvTasNet (2019) and did not benchmark against more recent methods such as Mossformer or Mossformer2.

Our main contribution lies in demonstrating the feasibility and flexibility of supporting end-to-end (E2E) speaker diarization, speech separation, and multi-speaker ASR tasks within a unified framework. To illustrate this, we adopt ConvTasNet, a widely recognized open-source speech separation model that operates in the time domain. Our experiments show that integrating ConvTasNet with speaker diarization and multi-speaker ASR tasks in a multi-task configuration improves its performance compared to training ConvTasNet as a standalone model. We believe this approach is not limited to ConvTasNet and could similarly enhance other speech separation models, such as Mosformer or Mosformer2, if they become available as open-source models. This demonstrates the flexibility of our multi-task architecture and its potential to improve speech separation models when integrated with diarization and ASR tasks.

---

For multi-speaker ASR, the proposed method achieved a WER of 9.2% on LibriSpeechMix. In contrast, the state-of-the-art performance is 3.43%, as reported in "Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System."

We consider this paper "contemporaneous" based on the reviewer guide provided by ICLR (https://iclr.cc/Conferences/2025/ReviewerGuide [FAQ for Reviewers]), which states:

“Q: Are authors expected to cite and compare with very recent work? What about non-peer-reviewed (e.g., ArXiv) papers? (updated on 7 November 2022)
A: We consider papers contemporaneous if they are published within the last four months. That means, since our full paper deadline is October 1, if a paper was published (i.e., at a peer-reviewed venue) on or after July 1, 2024, authors are not required to compare their work to that paper. Authors are encouraged to cite and discuss all relevant papers, but they may be excused for not knowing about papers not published in peer-reviewed conference proceedings or journals, which includes papers exclusively available on ArXiv. Reviewers are encouraged to use their own good judgment and, if in doubt, discuss with their area chair.”

The non-peer-reviewed version of the referenced paper (https://arxiv.org/abs/2407.09817) was published on ArXiv on July 13, 2024, while the peer-reviewed version (https://www.isca-archive.org/interspeech_2024/meng24c_interspeech.html) was published on September 1, 2024. Given that our submission deadline was October 1, 2024, we did not include the results from this paper in our initial version as they fall within the "contemporaneous" period defined by the reviewer guide.

“Q: If a submission does not achieve state-of-the-art results, is that grounds for rejection?
A: No, a lack of state-of-the-art results does not by itself constitute grounds for rejection. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge. Submissions can achieve this without achieving state-of-the-art results.”

We thank the reviewer for the detailed feedback and constructive questions. We are encouraged by the recognition of the strengths of our work.

Below are the detailed responses to each query.

Weaknesses

The paper is well-structured, but there are some sentences which are not clear or contain typos.

We have improved the overall clarity of the paper by refining the sentences and correcting the spelling mistakes indicated by the reviewer xgGR.

Questions

Can the authors comment on the use of 100% overlapped speech for evaluation of SD? Does the SUPERB benchmark for SD also use 100% overlapped speech?

We follow previous studies [1,2] to evaluate the speaker diarization task using 100% overlapped speech in the simulated dataset. Furthermore, to the best of our knowledge, the SUPERB benchmark, as reported by its authors [2], focuses on a two-speaker scenario in the LibriMix dataset, which also consists entirely of 100% overlapped speech.

---

What model type was used for OWSMv3.1(base, small, medium , LR)? From Figure 3 I assume medium, but I do not think the text states this anywhere.

In this study, we employed "OWSMv3.1 medium," which will be explicitly mentioned in the manuscript.

What does a "flat start" mean in line 342? & For how many epochs/steps did you train in total?

During training in UME, we initialize the encoder parameters with the pre-trained OWSMv3.1 medium encoder and fine-tune the encoder layers for 70 epochs, while all task-specific parameters have a flat start (i.e., no parameter initialization for task-specific layers) and are trained for 70 epochs. For the ASR-initialized UME model, the multi-speaker ASR model is pre-trained separately for 30 epochs, and then the ASR-specific head in the UME model is initialized from this pre-trained model. This results in a total of 70 epochs of fine-tuning for the OWSMv3.1 encoder layers, 70 epochs of training for the diarization and separation tasks, and 70 epochs of fine-tuning for the ASR task.

We have added the above discussion in Section 4.3 of the revised manuscript.

---

How much VRAM is needed to train your model, which GPU(s) did you use, and how long did it take to train? & What batch size did you use, and how were these sampled?

Four A100 80GB GPUs are used during training, and the batch size is dynamically adjusted based on the input length using the numel batch type in the ESPnet toolkit. In our experiments, the average batch size was 44, and it took six days to train the model for up to 70 epochs.

We have added the above discussion in Section 4.3 of the revised manuscript.

---

What is the minimum, maximum, and average duration of utterances in your train/eval dataset(s)?

The training and evaluation sets have utterance durations with a minimum of 3.1 seconds, a maximum of 56.8 seconds, and an average of 16.2 seconds. Additional details can be found in the corresponding table and are further discussed in Section 4.1 and Appendix A.2 of the revised manuscript.

---

I think it is unexpected (From Figure 3 and Figure 4) that the weights are nearly identical between task heads. Do the authors know what weights are found when training in a single-task setup, and whether they are similar or different to the values displayed in these figures?

We have incorporated the weights from the single-task setups and observed that their weight distributions differ from those in the multi-task setups. However, the overall trend remains consistent: the initial and final layers contribute more in the single-task setups, except for the diarization task, where all layers contribute equally.

We have added these observations in the Appendix A1 of the revised manuscript.

---

References

[1] Chen, Sanyuan et al. “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing.” IEEE Journal of Selected Topics in Signal Processing 16 (2021): 1505-1518.
[2] Yang, Shu-Wen et al. “SUPERB: Speech processing Universal PERformance Benchmark.” Interspeech (2021).

I want to thank the authors for their clear reply and updating the work in such a short time-span. I believe the results and conclusions as-is are not relevant to the community due to the 100% overlapping speech in the SD task, so I will not update my score.

However, this also means that the SD task in SUPERB is not relevant, which I did not know before this review, and is something which I want to explicitly state is non-obvious and the authors should not blame themselves for having followed the SUPERB SD recipe.

We thank the reviewer for the detailed feedback and constructive questions. We are encouraged by the recognition of the strengths of our work.

Below are the detailed responses to each query.

Questions

Clarification on Figure 1: Figure 1 appears misleading. Equation (3) shows different weights for different tasks, yet Figure 1 depicts only a single weight. Can you clarify this discrepancy?

We have made the corrections in Figure 1.

---

Training with Real Data: How do you propose to use real data for training the model, given that datasets for these tasks (e.g., ASR and speech separation) are typically disjoint? For example datasets for ASR do not generally contain ground truth for speech separation. How will this disjoint nature impact the joint training of the model?

Thank you for raising this important point regarding the challenge of training with real data for the joint tasks of separation, diarization, and ASR. We acknowledge that, in practice, datasets for these tasks are often disjoint, with ASR datasets typically lacking ground truth for separation and diarization, and vice versa. Here is our approach to mitigate this limitation:

• We leverage synthetic or simulated datasets where ground truth for all three tasks (separation, diarization, and ASR) is available. This pre-training step enables the model to learn joint representations effectively and establish a foundational understanding of each task.
• Once the model is pre-trained on synthetic data, we fine-tune it using real-world data for each task independently, while keeping the other task parameters frozen. This strategy will enable the professionals to use real diarization, separation and ASR datasets to adapt to each component of the UME framework.

Similar strategies have been employed in existing studies to address the absence of ground truth for all three tasks in real-world datasets, e.g., as reported in [1] and [2]:

References

[1] Bredin, H. (2023) pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. Proc. INTERSPEECH 2023, 1983-1987, doi: 10.21437/Interspeech.2023-105
[2] Cornell, S., Wiesner, M.S., Watanabe, S., Raj, D., Chang, X., Garcia, P., Masuyam, Y., Wang, Z.-Q., Squartini, S., Khudanpur, S. (2023) The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios. Proc. 7th International Workshop on Speech Processing in Everyday Environments (CHiME 2023), 1-6, doi: 10.21437/CHiME.2023-1