Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems

Response to Claims and Evidence

1. Sort loss is not a complementary or supportive loss to the system. Only by including Sort Loss, the model learns to arrange speaker predictions in arrival-time order.

   a. If we only use PIL, the model does not have arrival-time sorting capability.
   b. We clearly state the goal of Sort Loss in the abstract: It is designed to bridge the gap between ASR tokens and diarization timestamps during “multispeaker ASR training”. And this is achieved by training the diarization model using Sort Loss or Sort+PIL hybrid loss.
   c. Therefore, it is fair to say rather PIL (in the hybrid loss) is playing a role as a supplementary loss since it additionally corrects the mapping between predictions and labels since sort-prediction could be erroneous in some cases.

2. The “minimal architectural adjustments” are clearly explained throughout the paper.

   a. Section 2.3, Section 3.1 and Figure. 2 Explain the downside of modular or pipelined multi-speaker ASR systems and we explain how our proposed method can train a multi-speaker ASR model without applying any specialized loss but using the standard token-level cross-entropy loss.
   b. If you want to visually check the “minimal architectural adjustments”, see “Sortformer” module and sinusoidal kernels in Figure 2. The original modules in the vanilla Carany model (FastConformer and Transformer Decoder) are not altered in terms of architecture. We optionally use adapters to boost the performance with less amount of trainable weights.
   c. In summary, our proposed approach maintains the original ASR model’s architecture, except the part where we inject encoder output with speaker-kernel injection.

Response to Theoretical Claim

1. We found couple of errors in the appendix F:

   a. In F2 and F3, all QW^{Q}, KW^{K}, VW^{W} should be changed to XW^{Q}, XW^{K}, XW^{W}.
   b. Regardless of the mistake a, we did not put a necessary condition “Q is learned parameters” to permutation invariance (See-[https://people.tamu.edu/~sji/classes/attn-slides.pdf]). Thus, the proof F.2 is incorrect and we will remove the proof of permutation invariance (F.2) in the final version. We do not use learnable query Q in our proposed system either.
   c. The Permutation Equivariance property (when there is no positional embedding) seems enough for supporting the need for positional embedding.

Response to Relation To Broader Scientific Literature

We believe that the reviewer’s claim “this may limit the contribution of the paper to multi-speaker ASR where there is no overlap between the speakers (and preferably, long pauses between them)” can hardly be considered a valid criticism, because:

1. The supplementary demo video clip is designed for reviewers to intuitively understand multi-speaker ASR tasks in action, not to assess the functionality and performance of the proposed system.

2. There is plenty of evidence that our proposed system works without such issue:

   a. More than half of the samples in the diarization datasets DIHARD3, NIST-SRE-2000, CH109, are longer than 5 minutes, and these datasets have a very frequent long gap between each speaker’s segment. In the multi-speaker ASR test set AMI Corpus, the overlap ratio is 14% and the rest of 86% speech is non-overlapping speech.
   b. Based on these reasons, the point raised by the reviewer’s claim “this may limit the contribution” doesn't seem like a reasonable concern.

Response to Other Strengths And Weaknesses

2. The additional computational cost could be discussed as follows:

   a. Training Time Impact vs pure PIL training
   ∙ Pure Sort Loss: +0.23% (17.038 vs 17.0 min/epoch)
   ∙ Hybrid Loss: +2.26% (17.385 vs 17.0 min/epoch)

   b. Inference Time of Multispeaker ASR: LibriSpeechMix test-3mix (2,620 files, total duration: 42,514.9s) on an NVIDIA RTX A6000, batch size 100, with 10-run averages.

   • MS-ASR with Sortformer: 300.213s
   • MS-ASR without Sortformer 297.891s
   • Adding Sortformer causes 0.78% overhead in runtime (1.0078x)

   c. Inference on standalone diarization: No added inference time compared to PIL trained models.

   d. Our view on additional computational cost:
   (1) Training Sortformer with Hybrid Loss (Sort Loss + PIL) increases runtime by only 2.26% (x1.0226) compared to PIL-only training, while Sort Loss alone adds just 0.23% (x1.0023).
   (2) For inference, standalone diarization (Sortformer) adds no extra time versus the PIL-trained model.
   (3) Multispeaker ASR inference time increases marginally by 0.78% (x1.0078) when using Sortformer.
   (4) On the LibriSpeechMix dataset, our system achieves a 25.6% relative error rate reduction (7.14% → 5.31%).
   (5) Given the significant performance gains, the added computational cost is quite negligible, making the claim of "significant computational cost" not very convincing.

3. The reviewer’s comment seems unfinished: “For example, the”.

Response to Claims And Evidence

1. Response to the reviewer's claim that the statement "Sort Loss solves the permutation problem is not fully supported":

a. What we cannot achieve without PIL+ alpha=0.5 is the "maximized diarization performance", not "resolved permutation".
b. Therefore, we believe we can say that "Sort Loss based training" resolves the permutation problem in speaker diarization training and multispeaker ASR training.
c. Even if we use only Sort Loss without PIL, the Sortformer still performs with the sorting capability and PIL is a helping hand.

2. WER/DER with Sort Loss

Showing the SOTA diarization with Sort Loss is not the contribution we claim.

3. Response to missing deeper analysis:

a. Given page constraints, we focused on demonstrating how Sort Loss enables diarization supervision using standard ASR token-level cross-entropy loss.

b. Table 1 shows all comparable systems exhibit similar DER degradation with more speakers, as each additional speaker introduces compounding diarization errors with the similar proportion as our proposed system. The sole exception (WavLM-EEND-VC) is not an end-to-end speaker diarization model, weakening the validity of direct comparisons.

Response to Experimental Designs Or Analyses

1. Alpha parameter tuning:

We did grid search on this alpha parameter then found that 0.5 is the best performing parameter. We will add the grid-search results in the appendix in the final version.

2. Runtime measurements:

a. Training Time Impact vs pure PIL training

• Pure Sort Loss: +0.23% (17.038 vs 17.0 min/epoch)
• Hybrid Loss: +2.26% (17.385 vs 17.0 min/epoch)

b. Inference Time of Multispeaker ASR:

LibriSpeechMix test-3mix (2,620 files, total duration: 42,514.9s) on an NVIDIA RTX A6000, batch size 100, with 10-run averages.

• MS-ASR with Sortformer: 300.213s
• MS-ASR without Sortformer 297.891s
• Adding Sortformer causes 0.78% overhead in runtime (1.0078x)

c. Inference on standalone diarization:

No added inference time.

Response to Other Strengths And Weaknesses

1. In the Table 1, we wanted to show:

a. Showing Sort Loss achieves SOTA is not the key contribution. b. Sort-Loss can perform at a comparable level with PIL-trained models.
c. Hybrid Loss outperforms PIL based method, without losing "learned arrival-time sorting capability".

2. Sort Loss shows no improvement over PIL

We believe this is a minor weakness because we clearly state the goal of Sort Loss in the paper. Resolve permutation during multispeaker ASR training bridging diarization timestamps and tokens. Improving the PIL method with Sort Loss is not the main contribution we claim.

3-1. Robustness:

One of our evaluation dataset, DIHARD3, as the name suggests "Diarization is Hard", includes 11 domains that are very challenging to diarize such as noisy restaurant conversations, web-video and street interviews. DIHARD3 is the most noisy and reverberant diarization evaluation dataset.

3-2. Low-resource language:

NIST SRE 2000 dataset includes multi-lingual speech (Mandarin Chinese, Vietnamese, Spanish, Tagalog, etc) and DIHARD3 also includes Arabic, Mandarin, Min Nan, Portuguese, Russian, and Polish language. Therefore, the evaluation on these datasets involve diarization performance on low-resource languages.

3-3. Missing evaluations

Hence, we do not think that the Sortformer diarization model is completely missing evaluations on noise robustness and low-resource languages.

Response to Other Comments Or Suggestions

1. We realized this margin crash only after the submission. We will fix this in the final version of the paper.
2. "A" in Equation (14) is the output state (also referred to as ASR embedding) from Fast Conformer encoder in Figure 2. We will specify this in the final version.

Response to Questions For Authors

1. “What is the loss used for Multi-speaker ASR Training Data?”

The answer to this question is “Cross Entropy Loss” and it is already clearly mentioned multiple times in the manuscript:

a. The last paragraph in Section 1 (Introduction) “multi-speaker ASR training ... tokenlevel cross-entropy loss”
b. Figure 2. See “Cross-Entropy Loss” at the top with the loss function symbol “L_{CE}”.
c. End of Section 2.3 “Our approach focuses ... based on token objectives and cross-entropy loss”
d. End of Section 3.3 “the model can be trained ... cross-entropy function..”
e. In Section 6 (Conclusion), “... thereby supporting cross-entropy loss-based training and unifying the multi-speaker ASR framework...”

2. Limitations of our proposed system:

a. Currently, our proposed system has a limited inference length of 45 seconds and limited maximum of 4 speakers in a session.
b. The implementation we use for the experimental results runs Sortformer and FastConformer encoder in serial manner. This adds up inference time to the total inference time although it is less than 1%.

Response to Experimental Designs Or Analyses:

Here is model size information for the models we listed in Table 1. We will add the model size to Table 1.

• MSDD: 31.1 M
• EEND-EDA: 6.4M
• WavLM-L + EEND-VC: 317M+
• EEND-GLA-Large: 10.7M
• AED-EEND: 11.6M
• AED-EEND-EE: 11.6M

Response to Other Strengths And Weaknesses:

2-1. Comments on speaker embedding experiments: We have done a few experiments by concatenating speaker embedding (TitaNet) for the same multispeaker ASR task. However, we were not able to obtain any improvements with both frozen and fine-tunable speaker embedding extractors. The reason we are making a guess is as follows:

a. The speaker embedding model is trained to distinguish 10K to 100K different speakers and the speaker representations are too complicated to make synergistic effect on 2~4 speakers in a session.

b. Not only speaker tagging accuracies, speaker embedding interferes with the ASR output states (ASR embedding from Fast Conformer Encoder) and damages the WER itself.

c. Since there is already plenty of content included in the paper, we did not include the speaker embedding experiments which have no improvements. In addition, concatenating speaker embedding for speaker adaptation or multi-talker ASR is not a novel idea, which has been already tried in many of previous studies. We did not think that showing speaker-embedding concatenation experiments bolster the main idea of this paper and the other contents should be prioritized to explain and demonstrate the proposed method.

2-2. Speaking of the speaker embedding model, Sortformer is initialized with the NEST (Nemo Encoder for Speaker Tasks) model that is self-supervised learned on large size unlabeled dataset and NEST can be trained to perform speaker verification/identification. In a way, we are already taking advantage of speech representation pretrained on large amounts of datasets.

3. The second section of the green speaker should still be spk3. Not only our system but this applies to all the speaker diarization systems or multi-speaker ASR. If you want to have a better understanding on multi-speaker ASR tasks, please watch the supplementary demo video that contains multi-speaker ASR in action.

Response to Other Comments Or Suggestions:

• We will update Figure 5 to intuitively explain the issues you highlighted in the final version.

Response to Questions For Authors:

1. Please see rebuttal for “Other Strengths And Weaknesses”.
2. Please see rebuttal for “Other Strengths And Weaknesses”.
3. We speculate that this happens because word-level objectives produce more gradient for speaker tagging than segment level objectives due to its token count.

a. The segment-level objective includes a speaker token whenever there is a speaker switch (i.e., speaker change), whereas in the word-level objective, each word is accompanied by a speaker token.

b. In the proposed MS-Canary system, we compute the cross-entropy (CE) loss on the multispeaker text output (consisting of both speaker and text tokens) for model training.

c. Consequently, the word-level objective places greater importance on correct speaker assignment in addition to accurate word recognition compared to the segment-level objective. This leads to an improved cpWER.

Side Note

Please also take a look at the other rebuttals for other reviewers regarding runtime measurements.

1. Response to Claims and Evidence

1-1. Robustness

While the reviewer may find our evaluation insufficient, we believe the robustness of Sort Loss is demonstrated through the datasets we used up to a certain degree:

• DIHARD3 ("Diarization is Hard") covers 11 challenging domains, including noisy and reverberant scenarios (e.g., restaurants, street interviews, web videos). To our knowledge, no other benchmark is more comprehensive for noisy and reverberant speaker diarization.

• LibriSpeechMix is specifically designed for overlapping speech evaluation, with overlap rates ranging up to 90%, including three-speaker overlaps. It is widely used in multi-talker ASR research.

• We selected widely accepted benchmark datasets to ensure reliable comparisons. Evaluating on untested datasets would compromise result validity. We encourage the reviewer to reconsider after reviewing the datasets' features.

1-2. Comment on NLU Task Evaluation

We do not think that evaluating multi-speaker ASR on NLU tasks is a critical omission:

• In our target applications (e.g., meeting transcriptions, patient-doctor dialogues, and real-time speaker-tagged transcriptions), the primary output (text with speaker labels) is often consumed directly. Thus, metrics like DER and cpWER are more relevant.

• Designing such an evaluation would require extensive methodology descriptions, which is beyond the scope of this 8-page paper. While NLU-based assessment is an interesting direction, we believe it warrants a separate study.

2. Response to Experimental Designs Or Analyses

"..LibriSpeechMix .. with fixed delays":

The reviewer’s concern about “fixed delays” is incorrect. LibriSpeechMix is artificially mixed, but each session uses randomized delays. No fixed delays are applied.

3. Response to Questions For Authors

3-1. Runtime Evaluations

Inference time on stand-alone diarization model:

Dataset: LibriSpeechMix test-3mix (2,620 files, total duration: 42,514.9s) on an NVIDIA RTX A6000, with 10-run averages.

Stand-alone diarization (batch size=1, collar=0.0, overlap included eval):

• Pyannote-Diarization-3.1 (most popular open-source diarization model):
  4m39.6s (RTFx=152.06), DER=0.2144, Speaker Counting Accuracy=0.4985

• Proposed Sortformer (123M params):
  2m14.6s (RTFx=316.01), DER=0.1346, Speaker Counting Accuracy=0.9763

Multi-speaker ASR (batch size=100):

• MS-Canary (170M params): 297.891s
• Sortformer-MS-Canary (293M params): 300.213s
  → Only 0.78% runtime increase (x1.0078) from adding Sortformer supervision.

3-2. Sorting Inaccuracies and Its Effect on Performance

The model does not make sorting errors; the model only makes errors in diarization:

• The Sortformer model does not make "sorting errors". We have never seen an example where Sortformer generates speaker segments’ arrival time in the wrong order. If trained with enough data and time, assigning speaker index in arrival time order is an easy task for the model.

• The error comes from “missed” or “false alarm” predictions—i.e., short segments missed or non-existing segments falsely detected. This type of error occurs in all diarization systems.

• Therefore, the starting time of the first speech segments of each speaker (t1, t2, t3, t4) are always in order: t1 < t2 < t3 < t4, even if the diarization is wrong.

• For this reason, we are not able to perform “sorting error-based analysis.” We can only evaluate diarization errors and cpWER.

3-3. Scenarios Where Sorting-Based Mapping Could Be Inaccurate

a. Training Phase

• Table 1 shows performance gaps between Sort-Loss-only and Hybrid models, revealing sorting-related disadvantages. Both PIL and Sort Loss have imperfections, but they complement each other (acting as mutual regularizers).

b. MS-ASR Training/Inference

• Errors typically occur when speakers begin with very short utterances (1–2 words).
• This increases cpWER through incorrect speaker assignments.
• Note: Such errors affect al diarization and MS-ASR systems, not just ours.

4. Relation to DOM-SOT and Comparison

• "Arrival time" is not the only quantity that can be used to resolve permutations. Other options include: Total speaking time (DOM-SOT), End time, Speaking rate can be used.

• However, only arrival time determines the permutation at the start and does not require future input. For example, using "total speaking time" requires input from start to end, unsuitable for streaming or divide-and-conquer scenarios.

• DOM-SOT work is not tackling the same problem. DOM-SOT proposes a different loss criterion for multi-talker ASR but does not investigate supervision of a speaker diarization model.