SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model

We thank the reviewer for appreciating our work and the time you spent on the paper. Your insightful suggestions have been invaluable in enhancing the quality and clarity of our paper. We hope that our response will address your concerns. We are happy to respond to additional questions during the discussion period.

1. About sound morphing preliminary in Section 3.1

We thank for the insightful suggestion about the clarity of the three criterions, we will revise this section with more details.

The three objective criterions are originally come from Caetano & Osaka (2012), which are not proposed by us (see l.67-68, 141-142, 533-534). Our contribution and motivation is to adapt a series of comprehensive quantitative metrics according to the criterions to further provide a formal evaluation system for future sound morphing comparison. We hope below can address your concern about the clarity of the three objective criterions.

Correspondence: According to Caetano & Osaka (2012): 'morphing requires a description of the entities being morphed (shapes, images, sounds, etc), followed by establishing the correspondence between these descriptions'. This criterion focuses on the semantic content (or description) of audio rather than perception. For example, in music morphing, the correspondence may ensure the morphed outputs have description in between source and target music, rather than an entire new music. In environmental sound morphing, semantic content may represent the class of environmental sounds.

Intermediateness: As in Caetano & Osaka (2012), the intermediateness criterion illustrates the morphed objects should be perceived as intermediate. The most straightforward way to evaluate the intermediateness through a morphed sequence, where if the perceptual difference between the current transition and the previous transition is large, the large perceptual jumps may cause the intermediate objects to deviate from their role as "in-between" states, violating the intermediateness criterion. The "in-between" state could be interpreted by both the cases, which are interpreted as conceptual and categorical perception by Caetano & Osaka (2012). Sound is different from images or shapes. Sound morphing can draw inspiration from visual techniques, such as smooth transitions and perceptual metrics, but must adapt these to the auditory domain.

Smoothness: The linear assumption assumes each transition should have the same amount of perceptual change (i.e., $\Delta p$) if change the same amount of morph factor (i.e., $\Delta \alpha$) (see l.146-147). This means the relationship between morph factor and morphed sound perception is linear (i.e., you can image a linear function where x-axis is morph factor and y-axis is morphed sound perception). It can be formulated as $x^{(\alpha_{i+1})} - x^{(\alpha_{i})} = p_{\alpha_{i+1}} - p_{\alpha_{i}} = C$, where $C$ is a constant.

2. Regarding the Mel scale and human perception.

We thank for this insightful suggestion. We actually used the log-scaled Mel spectrogram in Equation(7), which potentially solved the problem of imbalance change in low-frequency and high-frequency areas. We will clarify the connection of the Mel scale with the nature of human perception in our revised version, with proper references.

3. The comparisons of this paper are based on its own metrics.

Since existing works target on a specific sound morphing scenario, in which metrics they used only applicable under the certain scenario. According to Caetano & Osaka (2012), Gupta et al., (2023) and Kamath et al., (2024), existing sound morphing methods lack of a formal and comprehensive evaluation system. For example, MorphFader (Kamath et al., 2024)performs morphing by text prompt, which evaluates ‘smoothness’ by text-audio similarity scores based on CLAP. MorphGAN (Gupta et al., 2023) performs audio texture morphing by interpolating parameters in an audio classifier, in which calculates parameter sensitivity and distribution similarity.

Instead, our motivation is to address this problem and make a formal sound morphing evaluation system that may benefit to future comparison. Furthermore, the comparison with MorphFader already showcases its advantage of flexible and formal evaluation, even if MorphFader doesn't release their codes, we can still make a fair comparison with the provided morphed results.

4. Detail formulation for computing FAD and FD in the experiment.

The calculation of FD and FAD are similar. Given two sets of audio data, $X_{morph}$ and $X_{sourced}$, where $X_{morph}$ contains consecutive morphed results as $X_{morph} =\\{x^{\alpha_i}\\}^N_{i=1}$, and $X_{sourced}$ contains sourced audios in that class.

Step 1. Calculate feature embeddings by a pre-trained audio classification model (i.e. VGGish for FAD and PANNs for FD), where embeddings capture high-level features of the audio.

Step 2. Calculate the mean $\mu$ and covariance matrix $\Sigma$ of the feature embeddings for both $X_{morph}$ and $X_{sourced}$. As $X_{morph} \sim N(\mu_{morph},\Sigma_{morph})$ and $X_{sourced} \sim N(\mu_{sourced},\Sigma_{sourced})$.

Step 3. metric = $|| \mu_{sourced} - \mu_{morph}||^2 + Tr(\Sigma_{sourced} - \Sigma_{morph} - 2\sqrt{\Sigma_{sourced}\Sigma_{morph}})$

Since the mean and covariance matrix captures distribution information of the morphed and sourced audio sets. Therefore, this metric helps to evaluate semantic consistence between morphed results and sourced samples. The smaller value of the metrics represents they are in a closer distribution level. In our experiment, we use the package from https://github.com/haoheliu/audioldm_eval to obtain the two metrics.

5. Could you provide justification for why CDPAM is a relevant metric?

CDPAM is trained to align with human perception, focusing on exploring higher level of perceptual audio features. Music and environmental sounds, like speech, rely on perceptual qualities (e.g., timbre, rhythm, spatial texture) for evaluation, which makes CDPAM conceptually extendable. Although CDPAM is evaluated on speech related tasks in their paper, its focus on contrastive learning of perceptual features often translates well to other domains where such features are important. For instance, music involves harmonics and temporal dynamics, similar to speech prosody. Environmental sounds rely on texture, temporal patterns, and frequency richness, overlapping with features modeled by CDPAM.

There are some references that use CDPAM for evaluating environmental sounds and music data:

[1] Hai, Jiarui, et al. "Dpm-tse: A diffusion probabilistic model for target sound extraction." ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.

[2] Jacobellis, Dan, Daniel Cummings, and Neeraja J. Yadwadkar. "Machine Perceptual Quality: Evaluating the Impact of Severe Lossy Compression on Audio and Image Models." arXiv preprint arXiv:2401.07957 (2024).

6. Lack of comparison with baselines for timbral morphing, environmental sound morphing and music morphing.

Please see Q3 in official comments and Q3-5 for the rebuttal response of Reviewer HAc4.

Even though SMT is an out-of-date timbral morphing method; however, we can't find any other open-sourced related methods to make comparison in the task of timbral morphing. As we mentioned in l.40-41, the sound morphing methods based on traditional signal processing techniques limits application for inharmonic sounds. Therefore, it is not surprising that the results from SMT are not performing well on complex music compositions. We also found SMT fails in the case of Taiko-Hihat in our experiment (see l.366-367).

Unfortunately, we cannot find any ML-based similar methods mentioned in our related work section with official implementation codes for a direct and fair comparison for the three tasks. We tried our best to find a fair and direct comparison with other existing methods, that is the reason why we also compared with a concurrent method based on examples from their demo page even though they haven't released their codes yet.

If the reviewer has any suggestions about the baseline methods, please elaborate.

7. Model comparison with the concurrent work

We thank for the suggestion from the reviewer, unfortunately, MorphFader hasn't released their implementation codes. We cannot do more rather than making direct comparison with the 7 samples on their demo page. We provide further explanation about this model comparison as well as a visualization comparison in Appendix 10. As we claimed in Appendix 10, we cannot compute FAD, FD, and CDPAM$_{E}$ value since the demo page doesn't provide actual sourced audios rather than reconstruction audios when $\alpha= 0$ and $\alpha =1$.

8. Regarding MOS study

Due to the lack of direct fair comparable baseline method in sound morphing research, this MOS study aims to validate the robustness of SoundMorpher across two different real-world applications, and further emphasis the robustness of a generalized sound morphing method.

9. Typos and the repeated notation (i.e., $N$ in PCA) in the paper.

We sincerely thank for the careful review, we will correct the these accordingly.

We extend our sincere thanks to the reviewer for dedicating their time to review our paper. We hope our response can address your concerns and make it clear to you on our motivations and contributions. We are happy to respond to additional questions during the discussion period.

1. The paper lacks novelty, and it is a straightforward adaptation of IMPUS to the audio domain.

Please see Q1 and Q2 in the official comment.

2. The choice of L2 over mel-spectrograms may be less appropriate

As we explained in l.257-262, Equation(7) elegantly solves the challenges we mentioned in l.252-254. This solution is also been agreed by Reviewer 3zUQ as well has been proved by our ablation study in Section 5.5. If the reviewer insists the Mel-spectrogram in Equation(7) is less appropriate, please elaborate.

3. Why not using CDPAM as the perceptual metric?

As we claimed in Section 3.1 as well as in l.258-259, the desired sound morphing results should not only ensure perceptual intermediateness and smoothness, but also should ensure correspondence. These three criterions are proposed by Caetano & Osaka (2012) to formalize the evaluation of sound morphing. Although CDPAM provides perceptual distance between two audios, we cannot ensure the $x^{(\alpha_i)}$ in-between $x^{(0)}$ and $x^{(1)}$ if using CDPAM as the metric in Equation(7). This violates the intermediateness criterion. Additionally, CDPAM doesn't provide semantic (e.g., audio description) information of audio, which cannot ensure the correspondence criterion. Rather, Mel-spectrogram provides both perceptual information and description of an audio, which is a better solution for the three criterions.

This strongly highlights that our method is not merely a straightforward adaptation of IMPUS but incorporates meaningful insights specific to the sound morphing domain.

4. Concatenate z-space in Equation(8).

This is an insightful idea. Concatenating in the z-space for dynamic morphing indeed produces better audio quality compared to concatenating waveforms. While waveform concatenation does not exhibit obvious abrupt transitions and offers advantages such as controllability in the time dimension, we will revise Equation (8) to provide the alternative solution of z-space concatenation. We thank for this valuable suggestion.

5. There are some missing references concerning controlled interpolations using audio diffusion models.

We tried our best to provide a comprehensive literature review for the sound morphing task. This is also been agreed by Reviewer 3zUQ. If the reviewer has any suggestions, please elaborate.

6.Provided audio examples are not particularly convincing and transitions sound abrupt, especially with as few points as five.

As we mentioned in our demo page, "Our demonstration also includes failure cases with abrupt transitions for SoundMorpher that two input sounds as significant semantic differences in content" to further illustrate the limitation that we observed in Appendix 11.4 (please also see l.522 - 525 in main text).

7. Where do the "Static morphing; Cyclostationary morphing & Dynamic morphing" terms come from?

The three morph methods are commonly used in the sound morphing, as we introduced in Section 3.1. We humbly suggest the reading group video recording of sound morphing in https://www.youtube.com/watch?v=fI8Wqc7e3Zk (start from 26 min 30 sec) may help the reviewer to understand the three sound morphing methods.

8. The choice of the interpolating path is pretty arbitrary, and the presented technique could be applied over any path.

We don't understand the meaning of "arbitrary". If we have misunderstood, we kindly ask for clarification for this review to ensure we address your concern accurately.

We interpolate conditional embeddings via a linear interpolation (l.232) as the conditional embeddings in AudioLDM2 are semantic structured, and we use spherical linear interpolation (l.238) to interpolate latent states following by Song et al. (2020). We perform perpetual uniform sound morphing by interpolating N points between $p^{(0)}$ and $p^{(1)}$ to ensure $\Delta p$ is constant in the morph path (l.269 - 282).

9. Is the perceptual metric always monotonic over these trajectories?

The proposed perceptual metric SPDP strictly and always monotonic over the morph trajectories, since it obtained by calculating the distance proportion between the Log magnitude Mel-spectrogram between source and target audio as in Equation(7). Given a SPDP point $p_i = [\tilde{p}_i^0,\tilde{p}_i^1]$, the summation of $\tilde{p}_i^{0}$ and $\tilde{p}_i^{1}$ is always equal to 1. Please also see Figure 8 - Figure 10 in Appendix 12 for a strongly visual evidence, where the spectrogram of morphed results are strictly in between source and target.

10. The experiment mainly compares with SMT.

Please see Q3 in the official comment.

We extend our sincere thanks to the reviewer for dedicating their time to review our paper. We hope our response can address your concerns and make it clear to you on our motivations and contributions. We are happy to respond to additional questions during the discussion period.

1. Regarding contribution and novelty

Please refer to Q1 in the official comment and Q3 in the response of reviewer kD4m. Our challenges are not only tuning a model and designing perceptual metrics; however, as we claimed in Section 3.1, sound morphing is inherently different from image morphing. Sound is more abstract than image and has various application scenarios, as our experiment illustrates.

2. Regarding three criterions that similar to IMPUS

Please refer to Q2 in the official comment.

3. Regarding the baseline with poor performance for timbral morphing

Even though SMT is an out-of-date timbral morphing method; however, we can't find any other open-sourced related methods to make comparison in the task of timbral morphing.

As we mentioned in l.40-41, the sound morphing methods based on traditional signal processing techniques limits application for inharmonic sounds. Therefore, it is not surprising that the results from SMT are not performing well on complex music compositions.

If the reviewer has any suggestions for a fair baseline comparison, please elaborate.

4. Regarding the MOS study with a small number of assessors

We obtained 21 completed survey answer sheets for the MOS study, which is a fair number according to other related works such as Kamath et al. (2024) (i.e., 18 assessors) and Zhang et al. (i.e., 26 assessors). Since there is no direct fair baseline methods for comparison, this MOS study aims to validate the robustness of SoundMorpher across different types of sounds, and further emphasis the robustness of a generalized sound morphing method.

5. Regarding the comparison with the concurrent work

We thank the insightful suggestion from the reviewer, we have already provided detail explanation for this comparison in Appendix 10 as well as visualization of spectrogram comparison in Figure 5. Again, what we want to emphasize is that the comparison is based on a concurrent work with improvement. Please also refer to Q4 in the official comment.

6. Regarding MFCC$_{\mathcal{E}}$ metric

The MFCC$_{\mathcal{E}}$ metric measures content consistency (i.e., the correspondeness criterion proposed by Caetano & Osaka (2012)) in the midpoint of morphed sequences and is strictly obtained by the equation we provided in Equation (15). We didn't take an average, but we obtain the value by calculating the absolute error between the proportion of how the midpoint result with $x_s$ with 0.5. Since it's a proportion metric not a similarity, the summation of the morphed result between $x_s$ and $x_t$ is always equals to 1.

7. Minor issues: “Spectral contras” p.9:450; “Mean opinion socre” p.20:1034

We will correct the typo accordingly, thank you for the carefully reviewing.

8. Did you try to apply your method to real music recordings?

We thank for the insightful suggestion. Real music recordings may encounter problems such as copyrights and recording qualities. Instead, we will release our codes upon the publication for interested users. Evaluating on high-quality synthesized music is a commonly used method for music-related research, e.g., Zhang et al. (2024) and Kemppinen P., (2020).

We provided a visualization of applying our method to complex sound scenes from AudioCaps in Figure 10. As well as the comparison with the demo examples from the concurrent work, MorphFader, is based on recordings from AudioPairBank. Due to the size limitation for the supplement, we cannot provide those in our demo page.

This already sufficiently showcased our method is applicable in the real-world applications.

p3. "If the reviewer has any suggestions for a fair baseline comparison, please elaborate."

Sure. In this case, you might have reproduced any of the sota papers in your domain. It is, of course, not an easy task but it would make your results look stronger. In the current situation, the reviewer cannot properly assess your results as the authors: 1) don't have propoper baseline; 2) don't provide convincing demo samples.

p4. Lack of baseline and anchoring (e.g. adding a deliberately bad and/or gt recording) given the difficulty of the concept of 'intermediateness' makes the MOS scores practically meaningless. It doesn't tell us that the model performs 'equally well' in both cases but that the score is volatile and random (see variance).

p6. MFCC is a decorrelated and compressed (PCA-like) representation of the mel-spectrum. I really don't understand what you mean by content here.

We thank for the reviewer's feedback and the effort you spent on our paper. We wish below response can further clarify our paper. We are happy to respond to additional questions during the discussion period.

1. To reproduce any of the SOTA papers in your domain for a comparison.

We appreciate the reviewer's suggestion; however, we respectfully disagree

(1) Even though the lack of direct and open-sourced comparison in the sound morphing area, we will release our implementation codes upon the publication, which our paper will be the first to provide a fair and direct comparison for future sound morphing studies.

(2) We have already compared with a concurrent work, which is superior to any public SOTA methods, this comparison already showcases our method is even superior to the concurrent method.

(3) Reproducing other SOTA papers is a solution; however, it is not a fair comparison. The papers may not offer enough detail for reproducing, and the dataset they used is not always accessible. As the reviewer suggests, low-quality results are not convincing.

2. Don't provide convincing demo samples and add a deliberately bad recording.

We thank for the reviewer's feedback; however, we humbly and respectfully disagree with this. We emphasize that, as we claimed in our demo page, our demo samples are without cherry-picking. The abrupt samples also provided on the demo page and discussed as limitation in Appendix 11.4. We don't understand what does the 'adding a deliberately bad recording' means. Both the baseline and the dataset we used are open public. The baseline method is open-sourced in https://github.com/marcelo-caetano/sound-morphing, and we have provided data source details in Appendix 13.

3. MOS study does not tell use the model performs equally well.

We have provided how the MOS study designed in Appendix 9, we designed this subjective study to verify the morphed results according to the three criterions. Although the variance of MOS score is relative higher, sound morphing is different from speech and sound generation tasks, as the sound morphing has no ground truths. This may cause the subjective study with higher variance; however, the mean of MOS scores for the two tasks are at the similar level. In addition, our demonstration also provides evidence that our method is successfully applied to either music morphing or environmental sound morphing.

4. Regarding the MFCC metric.

The MFCC$_{\mathcal{E}}$ metric verifies the correspondence criterion, detail explanations are provided in Appendix 8.2. MFCC is a higher level of audio feature derived from the Mel-spectrogram, which can reflect audio description information. This metric evaluates how the descriptions of the midpoint morphed audio result in-between two end points (see l.142 - 144, the definition of the correspondence criterion).