Solving Blind Non-linear Forward and Inverse Problem for Audio Applications

Thank you for your constructive and kind feedbacks. We appreciate the detailed comments you have made. For the primary revisions, please refer to the general comment we have made.

Q1.Please provide evidence for the claim near line 49 “that generative models are inferior to discriminative models in terms of quality”.

• We have removed this statement, recognizing it was overstated. The original intention was the state-of-the-art speech enhancement on various benchmark test set have been from discriminative rather than generative. However, considering its flexibility and generality, we conclude that the statement is not fair and overstated.

Q2. Explain the meaning and relevance of the statement on line 57 that particle filtering techniques reduce the 'hallucination effect'

• The hallucination effect refers to generative models producing artifacts or unrealistic outputs that deviate from the original signal’s characteristics. Since particle filtering is expected to better approximate the posterior distribution, the enhanced outcome is more coherent to the observed noisy speech, in terms of $y^* \approx \mathcal A(\tilde x)$. However, we removed the statement since we did not quantify or examine the effect on the hallucination effect in detail in the main text.

Q3. In Definition 1, if a pure noise-adding operator (as might be expected in speech enhancement tasks) is unbounded, is it excluded from the forward operators considered in this paper?

• Yes, unbounded operators are excluded. We focus on additive noise with 'finite SNR levels' between -5 and 15 dB, ensuring the operator remains bounded.

Q4. Complete the explanation following '(Right)' in Figure 1

• We have revised all the incompleted statement in the paper. Thank you for pointing it out.

Q5. Clarify the definition of 'forward problem' mentioned on line 129.

• We have elaborated on the definition in Section 3.1, Definition 2. Briefly, the forward problem involves estimating the forward operator $\mathcal{A}$ from the observed signal $y^*$, where $y^* = \mathcal{A}(x^*)$.

Q6. Provide the definition of A_{\theta}^{\dagger} on line 145.

• The notation $\dagger$ was intended to represent the pseudo-inverse of $\mathcal{A}$, such that $\mathcal{A}^\dagger \mathcal{A}(x) = x$. To avoid confusion, we have replaced this notation with $\tilde{\mathcal{A}}_\phi$.

Q7. Define c_g and c_l mentioned on line 161.

• $c_g$ and $c_l$ are global and local conditions from the reference encoder, respectively. We detailed the explanation in the main text.

Q8. Explain what 'sequence length' refers to on line 188

• We represents $c_g$ as a single vector and $c_l$ as a sequence of vectors. The sequence length means the length of the vectors in $c_l$.

Q9. Demonstrate how the training objective in section 4.3 is derived from the objective function defined in section 3.

• The training objective in Section 4.3 minimizes the empirical 2-Wasserstein distance between the push-forward measure of the approximated forward operator and the reference distribution. The additional terms (e.g., adversarial and spectrogram losses) are added to enhance perceptual quality and enforce domain-specific constraints.

Q10.11. Provide the proofs for Proposition 1 and Theorem 1., result of the theorem 1

• We have added sketches and main ideas of the proofs in the main text and Appendix. Theorem 1 validates the flexibility and generalizability of our approach to handle diverse audio effects and degradations, given sufficient 'basic operators' comprising linear and pointwise nonlinear operators.

Q12. Describe the derivation method for m(x_t) and C(x_t) in equation (8) of section 6.4.

• These terms represent the mean and covariance of the posterior distribution $p(x_0|x_t)$. We refer to the results in the paper "Tweedie moment projected diffusions for inverse problems (Boys et al., 2023)" for the explicit derivation. Specifically, the posterior mean $m(x_t)$ is known as Tweedie's formula, equivalent to the denoising function $\hat{x}0 = D\theta(x_t)$ in "Elucidating the Design Space of Diffusion-Based Generative Models" (Karras et al., 2023). $C(x_t)$ is the second moment of the posterior distribution, expressed as the gradient of the score function.

Q13. Comparative Methods

• We have added descriptions of the comparative methods at the end of the main text for the Voicebank/Demand and Reverb-WSJ0 benchmarks. While the objective metrics are worse than the baselines, their perceptual quality is good.

Q14. Is it possible to provide audio samples in the experimental section, e.g., audio samples corresponding to the spectrograms shown in Figures 6-9?

• Yes, we have provided an extensive list of audio samples at the link mentioned in the general comment.

Thank you again for your constructive and detailed feedback.

Thank you again for your response to the revised versions. We address the concerns you raised.

1. Yes, using the twisted particle filter with the approximated operator ($\tilde{\mathcal A}_\theta$ in our notation) is one of the main methods proposed in this work.

We emphasize the following primary advantages and claim novelties:

• Combining and Generalization of the previous works by the approximated operator

Prior works on diffusion-based inverse problems can be categorized into two groups.

• First, inverse problems with a known operator [1, 2, 3, 4, 5]. These approaches usually factorize $\nabla_{x_t} \log p(x_t|y)$ into $\nabla_{x_t} \log p(x_t) + \nabla_{x_t} \log p(y | x_t)$ and approximate the latter term via "posterior sampling." When dealing with linear operators, some works factorize the matrix $y = Ax$ using Singular Value Decomposition (SVD) to find a subspace for solving the problem. Image inpainting is a primary application in this context since the "masked portion" is known a priori.

• Second, blind inverse problems: There are many works on blind inverse problems; however, they are 'quasi-blind' in terms of specifying the structure of the operator. For example, BlindDPS [6] and BUDDy [7] assume $y = k * x$ and estimate the parameter $k$ during the generation process. More generally, GibbsDDRM [8] assumes $y = H_\phi * x$ to cover general linear operators. However, these methods are not applicable in real-world speech enhancement settings, where presuming the kinds or structures of operators is infeasible due to the presence of complex degradation factors such as reverberation, noise, and filters.

Our method both combines and generalizes these prior approaches. By approximating the general forward operator in a zero-shot fashion during the generation process, we overcome the limitations of existing methods that rely on known or structurally specified operators. Furthermore, the 'twisting' component of the twisted particle filter corresponds to computing the guidance term in the first category of methods. Therefore, we assert that our approach is more applicable to universal speech enhancement problems and addresses challenges that have not been tackled by previous works.

• Application of Particle Filters to Speech Enhancement

Particle filters operate by running $N$ Markov chains simultaneously and resampling each particle after the prediction step based on their weights, which are closely related to how well each prediction aligns with the observation. We apply this technique to the speech enhancement task to improve the quality of generation—a novel application not explored in similar works.

• Addressing Non-Linear Inverse Problems Prior works using twisted particle filters in diffusion process are only dealing with the linear and known cases [8, 9, 10]. However, when dealing with the non-linear operator, the main techniques used in the prior works are not available. Since many degradation operators appeared speech enhancement are highly nonlinear or non-differentiable (audio codec), we address this by simply linearizing operator as treating it as a linear operator by leaving the first term of the Taylor expansion. The linearlized operator is used at sampling from the proposal distribution and calculating weights for each particle in Equation 6. We suggest the future works to incorporate the first derivative term to reduce the approximation error such as low-rank approximation or diagonal approximation suggested in [11].

2. We agree that merely stating "the perceptual quality is good" is insufficient and not appropriate. The average MOS measured by SQUIM, a reference-less speech quality assessment tool, is 3.77 for the Voicebank/Demand dataset and 4.18 for the WSJ0+Reverb benchmark, even though the objective metrics are poor. Upon reviewing the audio samples we provided, we noticed that the clean audio examples contain inherent noise and are band-limited. Our model attempts to enhance these aspects as well. We will update our results with models trained only on matched datasets for the benchmark to provide a fair comparison.

Again, Thank you for your valuable advice on this matter.

3. Sorry for the inconvienience. we mistakenly omitted that part in the appendix. We have updated the proof in the appendix.

We appreciate the reviewers' insightful comments and believe that our revisions address the concerns raised. Thank you for your consideration.

We sincerely appreciate your advisory and valuable feedback. The most important revisions are presented in the general comments along with the audio samples. Below, we address your main concerns:

Irreversible Transformations and Restoration Quality

• As you pointed out, an inverse mapping does not generally exist for the irreversible transformations in a deterministic sense. Therefore, we have defined our inverse problem within a probabilistic framework. Despite the irreversible nature of some transformations, our objective metrics in Tables 1 and 3, along with the perceptual quality of the recovered signals, demonstrate that fine details can be effectively restored.

Importance of Perceptual Validation, Limited Comparative Evaluation and Reproducibility

• We have included results from subjective listening tests for the blind forward problem in Table 1. Additionally, we have provided audio samples for each experiment to facilitate perceptual validation. Please refer to the general comments for links and further details.

Q1. How does the proposed method compare to task-specific baselines in terms of objective and perceptual quality?

• Unfortunately, task-specific baselines were not examined, as they typically use different training datasets, such as instrumental or music datasets, or operate at lower sampling rates (e.g., 16 kHz). As a result, we found direct comparisons to be unfair. Instead, we evaluate various configurations of our proposed method and provide audio samples to facilitate qualitative assessment of the results.

Q2. Could the model incorporate assumptions about the original signal distribution to enhance the robustness of the restoration, and would this improve restoration quality?

• Our framework already leverages a probabilistic formulation, where the original signal distribution corresponds to the distribution of the input signals. This probabilistic approach inherently incorporates assumptions about the signal distribution, contributing to the model's robustness in restoration tasks. While we trained our model on a single dataset and tested it on synthetic data, we acknowledge that training on multiple and more diverse datasets can further improve robustness and generalization, especially in real-world speech enhancement scenarios.

Q3. In the context of semiring theory, could the properties (like associativity and distributivity) of operations in the DAG be explicitly defined to enhance computation or generalization?

• We have updated Section 3 to emphasize the advantages of constructing arbitrary forward operators. Properties such as associativity and distributivity allow the forward operator to be rendered in polynomial time by factorizing terms. Using a Directed Acyclic Graph (DAG) representation, the input signal can be processed efficiently via a topological sort. To ensure clarity and avoid excessive theoretical complexity, we have restated the relevant terminologies with concise and sufficient explanations.

Q4. To improve validation, could a listening test be added to evaluate the perceptual acceptability of restored audio, especially in cases where irreversible transformations have introduced ambiguity?

• We provide audio samples. Please refer to the general comment to find the url.

Thank you again for your constructive comments.

We sincerely appreciate your advisory and valuable feedback. The most important revisions are presented in the general comments along with the audio samples. Below, we address your main concerns:

• We have added comparisons with prior works in the experimental section and discussed them accordingly. However, since our model is trained exclusively on full-band audio from the VCTK dataset, it exhibits different enhancement behaviors. Nevertheless, the perceptual quality of our results remains comparable.
• As you suggested, we have added detailed information about the model in the appendix. Furthermore, we have publicly released the code to encourage further studies using our proposed method.

Q1. In Definition 1 (Forward Operator), it is assumed that both $\mathbf{x} \in \mathbb{R}^N$ and $\mathbf{y} \in \mathbb{R}^N$, where $N$ is the signal length. Does it mean that the proposed framework is only applicable to the case where the input $\mathbf{x}$ and output $\mathbf{y}$ have the same signal length $N$? Could you provide further details on how the proposed framework can be potentially extended to other inverse problems where the input and output signals may have different dimensionality?

A1. Yes, in our work, we assume that both the input signal $\mathbf{x} \in \mathbb{R}^N$ and the output signal $\mathbf{y} \in \mathbb{R}^N$ have the same fixed dimensionality $N$. This assumption simplifies two key aspects:

• With fixed input and output dimensions, the composition of operators is well-defined without the need to consider varying signal domains. This avoids cumbersome theoretical considerations related to changes in dimensionality during operator composition.
• We utilize U-Net as our main architecture, which is designed for inputs and outputs of the same size without padding. This choice simplifies the implementation and training process.

To extend our framework to the problems where the input and output signals have different dimensionalities, we need to address these challenges. We suggest some ideas to address these obstacles as following:

• Instead of using U-net that require fixed input and output sizes, we could employ sequential modeling techniques capable of handling variable-length sequences such as Transformer, VQ-VAE, RQ-Transformer variants, or state-space model (e.g. S4). based on the neural codec (e.g. Encodec)
• We may track the dimensionality every time the functions are composed. If we only consider the single-type effect, we can just address the problem by the architectural improvement.

Q2. In eq. (1), what does $\mathcal A \in C(K)$ mean?

A2. It denotes $\mathcal A$ belongs to the continuous function from $K$ to $K$. We apologize for any confusion caused by this notation and have updated the main text to clarify this point.

Q3. At line 159, maybe directly using $\mathbf{x}$ for clean signal and $\mathbf{y}$ for processed signal makes it more clear as they refer to the same thing. This could also help the reader understand what $\mathbf{x}$ and $\mathbf{y}$ are in the subsequent sentence at line 161

A3. We updated that point following your suggestion.

Q4. In Figure 2, what are the upper-case $X$ and $Y$?It looks like they are the spectrogram representations of $x$ and $y$, respectively, but such information is missing. In addition, do you use the complex-valued spectrograms or just their magnitude components for training the model?

A4. In Figure 2, the uppercase letters $\mathbf{X}$ and $\mathbf{Y}$ represent the spectrograms of $\mathbf{x}$ and $\mathbf{y}$, respectively. We apologize for not stating this explicitly and have updated the caption accordingly. We use a dual-domain architecture that processes both the waveform and its spectrogram. A 1D U-Net handles the waveform, and a 2D U-Net processes the complex-valued spectrograms with two channels (real and imaginary parts). The outputs from both are combined before producing the final result. This approach captures effects more effectively since some are easier to detect in one domain than the other. For example, "Clipping" appears as simple thresholding in the time domain but introduces non-harmonic structures in the frequency domain. Our primary experiments showed that combining information from both domains improves performance.

Q5. In Table 3, what do "Mix", "Blind" and "Known" stand for respectively?

A5. We have updated Table 3 and its caption to clarify the terms. Briefly :

• (Mix) : The baseline metric between the clean and noisy (mixture) signals.
• (Cond.): A conditional diffusion model without particle filtering.
• (GT): A conditional diffusion model applying particle filtering with the ground truth (GT) forward operator during inference. (Approx.): A conditional diffusion model applying particle filtering with the approximated forward operator.

Thank you again for your constructive feedbacks !

Thanks to the authors for responding to my questions. My concerns have been partially addressed. I appreciate the authors' effort on revising the manuscript which has clearly improved from the initial version. The audio samples provided and added comparison to other speech enhancement baselines are also helpful. Based on that, I increased my score from 5 to 6. Still, more detailed complexity analysis is missing, which I suggest the authors to include in their future iterations of the paper.

We sincerely thank you for the advisory and valuable feedback. Most important and primary revisions are presented in the general comment with the audio samples. Below, we respond to your main concerns:

• While we acknowledge similarities between our approach and Rice (2023) in detecting and sequentially removing audio effects from output signals, there are key differences:
  • Detecting audio effects via classification can lead to degeneracy issues, where different effect combinations result in the same effect, and the classification model may treat alternative combinations as errors. Our approach circumvents this problem by modeling the entire forward operator as a functional form. We have elaborated on this in Section 5 as per your suggestion.
  • Rice (2023) addresses a finite set of effect compositions with a static, monolithic configuration, whereas our method accommodates a significantly broader range of effect combinations.
  • RemFX employs effect-specific removal models, limiting flexibility in handling diverse real-world scenarios—a limitation our approach overcomes.
• We attempted to compare our model's performance with RemFX as recommended. However, direct comparisons are infeasible due to differences in training datasets. RemFX focuses on audio effect removal for instrument datasets (e.g., VocalSet, Guitar Set, DSD100, IDMT-SMT-Drums), while our work centers on speech datasets. Instead, we provide comparisons on standard speech enhancement benchmarks. The audio samples on the benchmark test sets are in the link https://t.ly/dBUhF.
• As suggested, we have removed the terms "dry" and "wet" signals in Section 1 to avoid introducing terminology prior to their formal definition in Section 2.

Thank you again for your constructive feedback.