ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior

We thank the reviewer for the thoughtful and encouraging comments.

Q: I think the method is very good, but the authors could have done a lot more to strengthen the evaluation. The authors only test with 2 speakers from the WSJ mix, never showing a result with more speakers.

We sincerely appreciate the reviewer’s kind remarks and will work hard to strengthen the evaluation. During the rebuttal phase, we ran new experiments on a commonly used 3-speaker reverberant source separation dataset. The results are in our rebuttal to Reviewer Go2n, where we show that ArrayDPS performs surprisingly well. Demos for the 3-speaker separation case are also updated in the demo page https://arraydps.github.io/ArrayDPSDemo/.

Q: Also, I would like to see some real results captured on real-microphone data. All the WSJ mixtures are synthetic, and I am skeptical whether the linear approximation A will hold up extremely well in real-world scenarios where there may be other influences on the captured sound. I would really like to see at least one real-world, non-synthetic example captured on any kind of microphone array.

To validate our method in real-world conditions, we recorded 15 mixtures in a 7m x 4m x 2.7m room. Two volunteers speak simultaneously under weak environmental noise and the mixtures are recorded by 3 microphones. Demos and recordings are in https://github.com/ArrayDPS/ArrayDPSDemo/tree/main/demo_results_realworld and are updated in the demo site https://arraydps.github.io/ArrayDPSDemo/. These new real world results sound consistent with the earlier synthetic results reported in the paper.

Q: The writing is clear. I do wonder if algorithms 1 and 2 can be simplified to distill the essence of the key high level meaning with more details left in the appendix. Also figure 1 has a lot going on. I don't think all the formulas are needed in the figure, instead, the high level ideas should be sufficient and make it easier to follow.

Thanks for your suggestions! Reviewer Go2n also suggests simplifying Algorithm 1 by abstracting out the second-order Heun sampler in EDM [2]. We welcome the advice. For the figures, we will remove some unnecessary formulas to make it clearer.

Q: Overall, I like the paper, as it feels like it tackles the blind source separation with DPS in an elegant way. I think the main novelty is in the estimation of the A matrix to be used in DPS (otherwise it's just DPS applied to a linear mixture of sources). This could be conveyed a bit better. Also, since that is a big part, it would be nice to see how well the predicted A matrix aligns with some ground truths in synthetically rendered scenarios.

We are grateful for the reviewer’s encouraging comments and are excited to answer the question related to the predicted A matrix. In the demo page https://arraydps.github.io/ArrayDPSDemo/, we updated a section called Filter and Source Visualization, where we show 6 figures corresponding to 6 mixtures.

In each figure, the first row contains source 1 anechoic source signal $\tilde{s_1}$, the ground-truth STFT domain RIR filter $h_{1,1}$ (from anechoic source to reference channel), and the reference-channel reverberant source 1 $s_{1,1}$. Of course, the third signal is the convolution of the first two signals. Then the second row shows the same for ArrayDPS, i.e., separated virtual source 1 $\hat{s_1}$, the final FCP estimated filter $\hat{g}^1_{0\rightarrow1}$, and the final reference-channel source 1 output $\hat{s_{1,1}}$. These two adjacent rows allow direct comparison between ground truth and ArrayDPS. Similarly, row 3 and row 4 show results for source 2.

We can visually see the difference between the final FCP estimated filter and the ground truth RIR through these figures. Those two signals might not be aligned, but sometimes show similar structure on the spectrogram. On the other hand, the virtual source separated by ArrayDPS is more like the anechoic signal rather than the reverberant signal. We explain this with two reasons: First, the diffusion model is trained on mostly anechoic clean speech, hence the inclination to generate anechoic sources. Second, FCP performs much better when the input source signal is anechoic (see Appendix G, and our rebuttal to reviewer Go2n, about why we are using the virtual source model). This means outputting an anechoic source (instead of a reverberant one) would satisfy a higher likelihood.

We also updated our demo site’s 2 speakers separation section with virtual sources. We urge readers to listen to the virtual source samples and compare them with the final output and the ground-truth anechoic source. It appears that ArrayDPS is indeed accomplishing some dereverberation, however, we do not want to over-claim here and intend to verify this promise in future research.

[1] Karras, Tero, et al. “Elucidating the design space of diffusion-based generative models.” Advances in neural information processing systems 35 (2022): 26565-26577.

We thank the reviewer for the detailed and constructive comments. New results from real-world experiments and a 3-speaker dataset are discussed in the rebuttal to reviewer Go2n. Corresponding demos are updated in https://arraydps.github.io/ArrayDPSDemo/.

Q: GET_SCORE(.) in Algorithm 2 contains wrong arguments.

This is a typo, where $D_\theta$ should be removed and $\hat{G}^{k, \text{IVA}}_{1\rightarrow c}$ should be inside the parentheses. We are sincerely sorry and will fix it.

Q: The dataset relies on simulated RIRs, raising generalization concerns.

To validate our method in real-world conditions, we recorded 15 mixtures in a 7m x 4m x 2.7m room. Two volunteers speak simultaneously with weak environmental noise and are recorded by 3 microphones. Demos and recordings are in https://github.com/ArrayDPS/ArrayDPSDemo/tree/main/demo_results_realworld and are updated in the demo site.

Q: The "Max" model is impractical and not fair for comparison.

Thanks for pointing this out. We will remove it from the ranking highlight and mark those as impractical.

Q: The array-agnostic baseline model used is relatively outdated, and TF-GridNet-Spatial is merely a supervised model trained on diverse microphone arrays. Comparison with a more relevant array-geometry-robust model, such as VarArray [1], should be considered.

We understand the concern that TF-GridNet [2], although a very strong baseline, is not designed in an array-agnostic manner. However, we do not think this TASLP 2023 SOTA paper is outdated due to its inability to handle any number of microphone inputs. Training on 3-channel ad-hoc arrays could result in better performance when evaluated on the same 3-channel scenarios. Of course, this needs further research for validation as there is no existing array-agnostic version of TF-GridNet for direct comparison.

For the VarArray baseline, we are a bit unsure about it as: (1) it follows FaSNet-TAC [3], but only reports word error rate and does not compare with FaSNet-TAC, (2) recent work [4] shows that it performs worse than FaSNet-TAC, and (3) to the best of our knowledge, there is no open-source implementation.

In light of this, we directly trained an array-agnostic FaSNet-TAC on WSJ0-2Mix with a variable number of channels, and report the separation results below:

We understand that this is by no means a fair baseline as it's using a 2020 backbone network. We will further look into a TAC version of TF-GridNet to fully clarify this issue.

Q: The impact of IVA initialization on performance is significant but lacks analysis.

Reviewer KMQX also mentioned this, so we refer to our rebuttal to KMQX, where we report three ablations around the IVA initialization. We first compare ArrayDPS initialized by IVA-Gaussian and IVA-Laplace. The second ablation is for parameter $N_{fg}$ (Algo. 2 line 9), which is the initial number of steps to use the IVA initialized filter. The third ablation compares using warm initialization or not in Algorithm 1, line 8. We believe these are the most important parameters to analyze, as the first shows ArrayDPS's sensitivity to IVA performance (Laplace prior performs worse), the second shows at what sampling time we should stop using the initialization and let FCP learn the filter, and the last one shows the effectiveness of initializing the diffusion starting point with IVA.

Q: Reference formatting issues

Thanks! We realized there are inconsistencies and issues in the references. We will fix them.

Q: Related work should be placed earlier.

Thanks for the suggestion! We thought about this but made the call to move it to the end primarily because it would contain background material from multiple domains, including unsupervised separation, diffusion posterior sampling, and RIR estimation, etc. We thought a long related work may interrupt the flow of the paper.

Q: line 739 has an extra ")", "row1,2,3" → "rows 1, 2, and 3", "Table G" → "Table 3", "Algorithm 3.5" → "Sec. 3.5"

Apologies. We will fix all these typos.

[1] Yoshioka, Takuya, et al. "VarArray: Array-geometry-agnostic continuous speech separation." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.

[2] Z.-Q. Wang, et al. "TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation," IEEE/ACM TASLP, vol. 31, pp. 3221–3236, 2023.

[3] Luo, Yi, et al. "End-to-end microphone permutation and number invariant multi-channel speech separation." ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2020.

[4] Chen, Weiguang, et al. UniArray: Unified Spectral-Spatial Modeling for Array-Geometry-Agnostic Speech Separation. arXiv preprint arXiv:2503.05110, 2025.

We thank the reviewer for the detailed and constructive comments.

Q: The paper could strengthen its evidence by evaluating more diverse real-world scenarios or noisy acoustic conditions to demonstrate true generalizability. Have you considered evaluating ArrayDPS on realistic datasets involving more complex acoustic conditions (real recordings, background noise, and dynamic speakers)? Could this impact your algorithm's stability and robustness?

Thank you for the suggestion. We ran new experiments for the following scenarios: (1) Real-world mixture recordings in a 7m x 4m x 2.7m room, where 2 speakers speak simultaneously. The recordings are performed using 3 microphones and under weak background noise. (2) A 3-speaker dataset to test the separation performance of all 3 speakers. The real-world experiment maintains strong separation performance, while in the 3-speaker (synthetic dataset) experiment, ArrayDPS outperforms the baselines by a large margin (to our surprise). The main results are reported here: https://arraydps.github.io/ArrayDPSDemo/, and additional demos and recording files are available here: https://github.com/ArrayDPS/ArrayDPSDemo/tree/main/demo_results_realworld.

Q: Sensitivity to hyperparameters like diffusion schedule, IVA parameters, and RIR estimation settings?

Thanks for raising this question. We ran some new ablations for six important parameters, namely: (1) the diffusion scheduler’s starting noise level $\tau_{\max}$, (2) IVA filter guidance steps $N_{fg}$ in Algo.2 line 9, (3) likelihood guidance $\xi$ in Eq. 57, (4) causal FCP filter length $F$, (5) initialized with IVA-Gaussian or IVA-Laplace (row 1b and 1c in Table 1), and (6) whether the warm initialization in line 8 of Algorithm 1 is used. Since inference is expensive, we show SI-SDR only for the first 30 mixtures of the 3-channel SMS-WSJ’s validation set (we sample each mixture 5 times). The default parameters are $\tau_{\max}=0.8, N_{fg}=100, F=13, \xi=2$ and IVA-Gaussian and warm initialization are used. For results below, we change one parameter at a time.

We believe the proposed algorithm is robust to most parameters, although the result is only on a 30-mixture subset. To make this more conclusive, we will finish the ablation on a larger dataset.

Q: Computational complexity for real-time deployment scenarios and possible strategies for optimization.

Currently, ArrayDPS and all neural baselines in this paper are impractical for real-time deployment scenarios due to the non-causal processing and high computation requirements. However, we think there are opportunities to mitigate this hurdle. First, we can use diffusion distillation techniques to reduce sampling steps or even make it one step. Also, our algorithm can easily fit into the flow matching’s sampling ODE, which would be more efficient. To enable real-time processing, we need to modify the diffusion U-Net to be a causal architecture, and then update FCP and the likelihood calculation with an adaptive version; we expect this to be reasonably straightforward.

Q: The algorithm heavily relies on IVA initialization for stability and performance. This dependency somewhat reduces the claimed elegance and universality of the method.

We agree with this. Currently, ArrayDPS is sensitive to initialization, bearing similarity to the Expectation Maximization (EM) algorithm. We will add a section to discuss this and also give insights for future directions (1. Inducing a filter prior during sampling. 2. In early steps, maintain some uncertainty for the filter instead of greedily optimizing using FCP).

Q: The result STD (around ±1.2 dB) exhibits notable instability, potentially limiting reliability.

We respectfully disagree with this argument since our method is generative, which means that our method samples multiple plausible solutions as separation results. These solutions might differ at the sample level, but all sound plausible. We believe (±1.2dB) is a reasonable range for 5 samples (please verify this in our demo about how all 5 different samples sound correct, but have ±1.2dB fluctuation). However, without IVA initialization, ArrayDPS-D exhibits a fluctuation of ±4.6 dB, implying our algorithm lacks consistency without IVA.

Thanks again for your insightful observations.

We thank the reviewer for the detailed and constructive comments.

Q: The advantage of using a virtual source model is unclear and questionable.

Yes, we understand this concern now. We had tried to explain this but the explanations are scattered in the paper (lines 147-152 (column 1), line 259 (column 2), and then again in Appendix G). We will consolidate these explanations in one place. Briefly, our main reason to propose a virtual source model is as follows:

-- When FCP computes the relative RIR even with respect to the ground truth (reverberant) signal measured at one channel, FCP fails to reconstruct any other channels' sources perfectly (only 11.3 dB of SI-SDR as shown in Table 3).

-- The reason is the relative RIR filter, as mentioned in Eq. 3 (lines 98-99), involves an inverse filtering, for which the inverse may not exist, and may also make the relative filter non-causal. We discuss this in Appendix G (lines 1143-1144).

-- On the other hand, Table 3 row 1 shows that FCP computes the relative RIRs much better when the source estimate is the original anechoic source.

Hence, we proposed to use the virtual source model, hoping that we can sample the anechoic source, thereby leveraging the effectiveness of FCP. We apologize for the confusion; we will explain this in detail early in the paper.

Q: The term intractable is misused to describe the likelihood.

We think we understand the issue. In previous literature like DPS[1], $p(y|x_t)$ is usually viewed as intractable because it needs to integrate through $x_0$, or $p(x_0|x_t)$ is intractable because it needs to integrate through all intermediate variables. In our case, we viewed $p(y|x_0)$ as intractable because it needs to integrate through all possible operators on $x_0$. Perhaps this was the source of the confusion. We still think that the intractable term applies to our case, but it needs to be appropriately explained. We will modify the paper accordingly.

Q: The experiments were conducted with only two speakers. More complex scenarios should be tested.

We ran new experiments with 3 speakers on the spatialized WSJ0-3Mix dataset (reverberant) [2]. Below is our result table on the 3-speaker evaluation. We also recorded real-world mixtures for the demo. The demo page https://arraydps.github.io/ArrayDPSDemo/ now contains 3-speaker demo and real-world mixtures demo. More real-world demo samples can be found in https://github.com/ArrayDPS/ArrayDPSDemo/tree/main/demo_results_realworld.

For the 3-speaker WSJ0-3Mix dataset [2], we are pleasantly surprised that ArrayDPS-A outperforms supervised TF-GridNet-Spatial by a large margin. This shows ArrayDPS's potential in more complicated scenarios.

Q: Missing citation of BlindDPS[3].

This definitely should be cited. We were supposed to cite this in line 421, second column, when we discussed the blind inverse problem with DPS, but we mistakenly cited DPS again there. Sincere apologies, and thank you for pointing this out.

Q: Typos in line 211 (confusion for S), second column, line 158-164, second column (missing log), Eq.6 (minus sign), line 136-137, second column (Gaussian typo), Eq.13 typo, line 20 of algorithm 1 (wrong args in get_score).

Thanks! We will fix all these typos in the next version.

Q: Algorithm 1 is too complex and has 2 return statements.

Thanks for this suggestion. We will abbreviate algorithm 1 and abstract out the second-order Heun sampler. We will also return both signals in the end.

[1] Chung, Hyungjin, et al. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687 (2022).

[2] Mitsubishi Electric Research Laboratories. Deep Clustering. https://www.merl.com/research/highlights/deep-clustering

[3] Chung, Hyungjin, et al. Parallel Diffusion Models of Operator and Image for Blind Inverse Problems. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, No. 2, 2023.